BitShares Forum
Main => General Discussion => Topic started by: bitmeat on July 20, 2014, 04:56:52 am
-
Ok so the story is as goes. I was trying to send 180K in chunks to another user. 2 of the transactions were showing as pending for me, and I cleared the pending transactions, rescanned/restarted/etc. so they disappeared, and I "had" my balance again.
So when I do from the console I get only the following transactions between me and lowfee-10prcnt (2 transactions missing one for 10K and one for 20K):
2014-07-20T02:31:59 8169 krabbypatty lowfee-10prcnt 1.00000 BTSX ensuring it works. 0.10000 BTSX a0da271
2014-07-20T02:37:40 8202 krabbypatty lowfee-10prcnt 10,000.00000 BTSX 1st batch 0.10000 BTSX 41a6c3e
2014-07-20T02:54:15 8302 krabbypatty lowfee-10prcnt 10,000.00000 BTSX 2nd batch 0.10000 BTSX 7e638b1
2014-07-20T03:28:21 8499 krabbypatty lowfee-10prcnt 20,000.00000 BTSX 3rd batch 0.10000 BTSX 524dba1
2014-07-20T03:31:09 8515 krabbypatty lowfee-10prcnt 20,000.00000 BTSX 4th batch 0.10000 BTSX 1af8cea
2014-07-20T03:57:48 8661 krabbypatty lowfee-10prcnt 30,000.00000 BTSX 5th batch 0.10000 BTSX baff8cc
2014-07-20T04:26:56 8830 krabbypatty lowfee-10prcnt 20,000.00000 BTSX 6th batch 0.10000 BTSX 6eb27a0
On his side he has:
2014-07-20T02:33:17 8169 krabbypatty lowfee-10prcnt 1.00000 BTSX ensuring it works. 0.00000 BTSX a0da271
2014-07-20T02:37:35 8202 krabbypatty lowfee-10prcnt 10,000.00000 BTSX 1st batch 0.00000 BTSX 41a6c3e
2014-07-20T02:53:10 8284 krabbypatty lowfee-10prcnt 10,000.00000 BTSX 2nd batch 0.00000 BTSX 244f2da <<<<<<<<<<<<<<<<<<
2014-07-20T02:53:10 8294 krabbypatty lowfee-10prcnt 20,000.00000 BTSX 0.00000 BTSX 8f4cea8 <<<<<<<<<<<<<<<<<<
2014-07-20T02:54:14 8302 krabbypatty lowfee-10prcnt 10,000.00000 BTSX 2nd batch 0.00000 BTSX 7e638b1
2014-07-20T03:28:24 8499 krabbypatty lowfee-10prcnt 20,000.00000 BTSX 3rd batch 0.00000 BTSX 524dba1
2014-07-20T03:31:04 8515 krabbypatty lowfee-10prcnt 20,000.00000 BTSX 4th batch 0.00000 BTSX 1af8cea
2014-07-20T03:57:44 8661 krabbypatty lowfee-10prcnt 30,000.00000 BTSX 5th batch 0.00000 BTSX baff8cc
2014-07-20T04:26:55 8830 krabbypatty lowfee-10prcnt 20,000.00000 BTSX 6th batch 0.00000 BTSX 6eb27a0
How do I reconcile my account to actually acknowledge that?
It says I have X amount, when I try to move X amount to another account it says insufficient funds and that I have X - 30K available BTSX. Clearly the system knows this happened, but it doesn't reflect it in my account.
-
Bump! Come on Invictus guys! I'm willing to work with you to solve this.
toast was the only one who even attempted looking at this. I would like an official word from someone who gets what's going on. I've provided logs and dumps to toast.
-
Hey there... you can imagine we all have our hands full right now, but we certainly want to help you solve it.
-
Hey there... you can imagine we all have our hands full right now, but we certainly want to help you solve it.
Right. I'm actually mostly concerned that this could get a lot of people in trouble. Especially when this goes to exchanges. I think it should be somewhat higher priority.
Yes, I can imagine your hands are more than full, and it's Sunday. And you've done a great job. But I think this is more important than discussing how delegates work.
-
1) Did you put a memo in every transaction? ie: can I assume those should be unique?
-
Have you done a wallet_account_balance in the console to see if it's just a UI bug?
-
1) Did you put a memo in every transaction? ie: can I assume those should be unique?
I've sent the 2nd batch twice with the same memo, because I thought it didn't send, so I canceled pending and resent. However I had network issues at the time, I restarted the client, which by the way always crashes on Windows, and not sure if some vital piece didn't get written where it needs to. However the receiver has all the funds.
He is missing one transaction in the ledger, and I am missing 2 in mine. What I want to know is how does the number at the top of the UI get computer, where do I find it in the command line. And more importantly why is the ledger not reconstructed from the network on rescan?
-
Have you done a wallet_account_balance in the console to see if it's just a UI bug?
It's not just a UI bug, if you look at the post above there is a dump from wallet_account_transaction_history in the console.
-
Here is what is going on:
He has received all transactions, but when you "re-scan" you do not detect outgoing transactions.
So when you cleared pending transactions you deleted your knowledge of those transactions from your wallet. His transaction record is accurate, your balance is accurate, but your transaction record is missing those 2 transactions.
-
Here is what is going on:
He has received all transactions, but when you "re-scan" you do not detect outgoing transactions.
So when you cleared pending transactions you deleted your knowledge of those transactions from your wallet. His transaction record is accurate, your balance is accurate, but your transaction record is missing those 2 transactions.
Are you saying that unless I keep my local hard drive safe, I have no way of ever reconstructing the missing transactions in the ledger?
I mean if I export my wallet, then some transactions occur, then my hard drive dies , will I be able to re-create my account from scratch from that older wallet?
(This happened to me by the way and I lost 400 PTS during donation period, because I thought Qt PTS used the same change address, so my backup didn't work). I also hope your change addresses are deterministic.
-
Here is what is going on:
He has received all transactions, but when you "re-scan" you do not detect outgoing transactions.
So when you cleared pending transactions you deleted your knowledge of those transactions from your wallet. His transaction record is accurate, your balance is accurate, but your transaction record is missing those 2 transactions.
No it was not. Only the balance in the account (the total) was.
It is accurate now though, ~12 h later.
-
I think that we need to update the UI in the following way:
1) clear pending transactions should only hide the transactions (not remove them from the wallet)
2) When scanning the blockchain, we need to pick up transactions that are outgoing even if we lost information about who they were sent to... TITAN makes rescanning to restore history harder because it is just that private.
3) The balance shown for your account is based upon known spendable outputs and is the "true" balance after a rescan. We are working to make sure the ledger accurately explains the true balance.
-
I think that we need to update the UI in the following way:
1) clear pending transactions should only hide the transactions (not remove them from the wallet)
2) When scanning the blockchain, we need to pick up transactions that are outgoing even if we lost information about who they were sent to... TITAN makes rescanning to restore history harder because it is just that private.
3) The balance shown for your account is based upon known spendable outputs and is the "true" balance after a rescan. We are working to make sure the ledger accurately explains the true balance.
I see. Yes, even if it shows "lost encrypted transaction info" it's better than having discrepancies, which can be very confusing.
Also - and more concerning - there is a missing transaction in the ledger on the receiving side. I actually am not worried about my transactions missing, so long as he received them. I'm just trying to help here.
I'm thinking how would someone like BTER handle that? Will they just have receiving address, without an account name? Can they create unregistered receiving addresses - one per account, so that even if the tx info is lost, they still get the deposits?
-
There is a scalability issue with having many TITAN receive addresses because you must check every transaction against all keys and the cryptographic operation is expensive. So BTER should have 1 account that they receive with.
We have guidelines available for exchanges, but I need to find where the guys published it.
-
The transaction with no memo for 20K is missing on receive side... very interesting.
Does his ledger balance out with his account balance after a rescan?
-
The transaction with no memo for 20K is missing on receive side... very interesting.
Does his ledger balance out with his account balance after a rescan?
I've instructed him to do a rescan as well. I think he was able to send the balance, but never saw it in the ledger.
I think he is also missing the transaction with 10K one of the "2nd batch" transactions - there are 2 with the same memo, but he got only one - look at TX ID on the right
-
Correct, told me that he *does* have the correct balance and is only missing the transaction for the 10k btsx
krabbypatty must have contacted you regarding this.
I just want to add that while my incoming transaction are about ~130K my combined accounts balance is 140K -pretty much where it should be, except I have no way to tell who paid the 10K difference.
-
Here is what is going on:
He has received all transactions, but when you "re-scan" you do not detect outgoing transactions.
So when you cleared pending transactions you deleted your knowledge of those transactions from your wallet. His transaction record is accurate, your balance is accurate, but your transaction record is missing those 2 transactions.
Are you saying that unless I keep my local hard drive safe, I have no way of ever reconstructing the missing transactions in the ledger?
I mean if I export my wallet, then some transactions occur, then my hard drive dies , will I be able to re-create my account from scratch from that older wallet?
(This happened to me by the way and I lost 400 PTS during donation period, because I thought Qt PTS used the same change address, so my backup didn't work). I also hope your change addresses are deterministic.
All private keys we generate are deterministic and can be regenerated.... as long as you have registered all of your accounts you will have an easy time recovering your balance, but your detailed account history could be gone for ever. If you have unregistered accounts and do not know the public keys for them, then you will have to brute force scan with all of your keys and that could take an eternity (doable, but painful).
For this reason I recommend registering all accounts on the blockchain *or* backing up all public keys for the unregistered accounts you do create.
-
Here is what is going on:
He has received all transactions, but when you "re-scan" you do not detect outgoing transactions.
So when you cleared pending transactions you deleted your knowledge of those transactions from your wallet. His transaction record is accurate, your balance is accurate, but your transaction record is missing those 2 transactions.
Are you saying that unless I keep my local hard drive safe, I have no way of ever reconstructing the missing transactions in the ledger?
I mean if I export my wallet, then some transactions occur, then my hard drive dies , will I be able to re-create my account from scratch from that older wallet?
(This happened to me by the way and I lost 400 PTS during donation period, because I thought Qt PTS used the same change address, so my backup didn't work). I also hope your change addresses are deterministic.
All private keys we generate are deterministic and can be regenerated.... as long as you have registered all of your accounts you will have an easy time recovering your balance, but your detailed account history could be gone for ever. If you have unregistered accounts and do not know the public keys for them, then you will have to brute force scan with all of your keys and that could take an eternity (doable, but painful).
For this reason I recommend registering all accounts on the blockchain *or* backing up all public keys for the unregistered accounts you do create.
This should be posted some place more prominent!
-
Agreed.
-
I have been working on the transaction history and wallet rescanning code:
- https://github.com/BitShares/bitshares_toolkit/compare/bcd3d69772...7291385d48
- https://github.com/BitShares/bitshares_toolkit/compare/d43c3982a9...fc69f9efa1
- https://github.com/BitShares/bitshares_toolkit/compare/c09d3fecb1...f998c122ad
- https://github.com/BitShares/bitshares_toolkit/compare/23372db089...dc1d28ec71
- https://github.com/BitShares/bitshares_toolkit/commit/0e41da25a206518588e8b6deb5a5db3e0ad95b87
There were quite a number of problems with the wallet transaction scanning causing missing entries, incorrect balances, silently failing rescans, etc. In my testing so far, transaction history reconstruction works much better now. I am able to delete every transaction record (withdrawals, deposits, account operations, asset operations) from my wallet and it correctly brings them all back (sans any irrecoverable information due to TITAN). More testing is required but I think we might be able to fix your issues.
These fixes will be in the next release. You guys will have to do a full rescan (backup your wallet first, just in case) and we can see if it fixes your transaction histories.
-
Great! Thanks for posting links to code!
Question:
new_entry = false;
+ // TODO: We should probably really have a map of asset ids to amounts per ledger entry
+ if( entry.amount.asset_id == amount.asset_id )
+ entry.amount += amount;
+ else if( entry.amount.amount == 0 )
+ entry.amount = amount;
Will you check this logic. And also - are there tests associated with this code? I know it's a lot to ask for but we should have good integration tests that try all kinds of things.
E.g. have you thoroughly tested with multiple assets. Or able to simulate a missing transaction on the sending/receiving end, and see how the system behaves?
Thanks! Cheers! Good job!
-
Great! Thanks for posting links to code!
Question:
new_entry = false;
+ // TODO: We should probably really have a map of asset ids to amounts per ledger entry
+ if( entry.amount.asset_id == amount.asset_id )
+ entry.amount += amount;
+ else if( entry.amount.amount == 0 )
+ entry.amount = amount;
Will you check this logic.
I see that this will fail when transferring assets other than the base asset, as I must have realized at some level before relegating the thought to the TODO there. Good catch! Will think how to address this.
Transactions can be almost arbitrarily complex but to keep meaningful records in our simple wallet we are making strong simplifying assumptions for now. We also have the problem that everyone already has a wallet using a certain format for storing their existing transaction records.
And also - are there tests associated with this code? I know it's a lot to ask for but we should have good integration tests that try all kinds of things.
E.g. have you thoroughly tested with multiple assets. Or able to simulate a missing transaction on the sending/receiving end, and see how the system behaves?
No tests yet, but I intend to write them when there is some time. Still many other fires to put out at the moment.
I have not thoroughly tested with multiple assets / market functionality yet, as is clear from the problem above. Missing transfers of the base asset on sending and receiving side are correctly reconstructed, however.
I also forgot to mention that in the OP, you seem to have been affected by problems with transaction scanning combined with an issue that causes transactions to be stuck pending. This is a known issue but we are still tracking down the cause. Sometimes they get unstuck after some random period of time, so be cautious about resending transactions.
-
I have answerred that several times and it worked!
do this:
wallet_delegate_set_block_production ALL false
wallet_rescan_blockchain
wait 1-2 minutes (I think it does not matter )
wallet_delegate_set_block_production ALL true
-
Just a reminder for the reason of this:
TITAN transactions are costly and time consuming.. if you have a delegate running on the network you need to sign within 5-10 secs ... otherwise you miss your spot .. however if you have several 'other' accounts active TITAN scanning will probably not finish in time .. yet ... multithreading is on the way as BM told us somewhere :)
-
Liondani this didn't work for me. I still have discrepancies. This only fixes transactions that stay pending forever. But if I've lost information about pending transaction it's gone for good.
-
Liondani this didn't work for me. I still have discrepancies. This only fixes transactions that stay pending forever. But if I've lost information about pending transaction it's gone for good.
that must be a priority issue for the team I guess...
-
These fixes will be in the next release. You guys will have to do a full rescan (backup your wallet first, just in case) and we can see if it fixes your transaction histories.
Do you guys see any improvement using the latest version? https://github.com/dacsunlimited/bitsharesx/releases/tag/0.2.3
-
These fixes will be in the next release. You guys will have to do a full rescan (backup your wallet first, just in case) and we can see if it fixes your transaction histories.
Do you guys see any improvement using the latest version? https://github.com/dacsunlimited/bitsharesx/releases/tag/0.2.3
Great! Using wallet_account_history and I now see a correct balance with the missing transactions from before shown as UNKNOWN.
However in the UI the balance column in the transaction history is now completely blank. as in not even numbers are seen there.
I'm closing this issue. I hope exchanges don't ever get to miss a transaction, that would be unfortunate.