[RESOLVED] Missing 30K - Help troubleshoot this.

[bitmeat]

These fixes will be in the next release. You guys will have to do a full rescan (backup your wallet first, just in case) and we can see if it fixes your transaction histories.

Do you guys see any improvement using the latest version? https://github.com/dacsunlimited/bitsharesx/releases/tag/0.2.3

Great! Using wallet_account_history and I now see a correct balance with the missing transactions from before shown as UNKNOWN.

However in the UI the balance column in the transaction history is now completely blank. as in not even numbers are seen there.

I'm closing this issue. I hope exchanges don't ever get to miss a transaction, that would be unfortunate.

[vikram]

These fixes will be in the next release. You guys will have to do a full rescan (backup your wallet first, just in case) and we can see if it fixes your transaction histories.

Do you guys see any improvement using the latest version? https://github.com/dacsunlimited/bitsharesx/releases/tag/0.2.3

[liondani]

Liondani this didn't work for me. I still have discrepancies. This only fixes transactions that stay pending forever. But if I've lost information about pending transaction it's gone for good.

that must be a priority issue for the team I guess...

[bitmeat]

Liondani this didn't work for me. I still have discrepancies. This only fixes transactions that stay pending forever. But if I've lost information about pending transaction it's gone for good.

[xeroc]

Just a reminder for the reason of this:

TITAN transactions are costly and time consuming.. if you have a delegate running on the network you need to sign within 5-10 secs ... otherwise you miss your spot .. however if you have several 'other' accounts active TITAN scanning will probably not finish in time .. yet ... multithreading is on the way as BM told us somewhere :)

[liondani]

I have answerred that several times and it worked!

do this:

wallet_delegate_set_block_production ALL false

wallet_rescan_blockchain

wait 1-2 minutes (I think it does not matter )

wallet_delegate_set_block_production ALL true

[vikram]

Great! Thanks for posting links to code!

Question:

Code Select Expand


new_entry = false;
+                     // TODO: We should probably really have a map of asset ids to amounts per ledger entry
+                     if( entry.amount.asset_id == amount.asset_id )
+                         entry.amount += amount;
+                     else if( entry.amount.amount == 0 )
+                         entry.amount = amount;


Will you check this logic.

I see that this will fail when transferring assets other than the base asset, as I must have realized at some level before relegating the thought to the TODO there. Good catch! Will think how to address this.

Transactions can be almost arbitrarily complex but to keep meaningful records in our simple wallet we are making strong simplifying assumptions for now. We also have the problem that everyone already has a wallet using a certain format for storing their existing transaction records.

And also - are there tests associated with this code? I know it's a lot to ask for but we should have good integration tests that try all kinds of things.

E.g. have you thoroughly tested with multiple assets. Or able to simulate a missing transaction on the sending/receiving end, and see how the system behaves?

No tests yet, but I intend to write them when there is some time. Still many other fires to put out at the moment.

I have not thoroughly tested with multiple assets / market functionality yet, as is clear from the problem above. Missing transfers of the base asset on sending and receiving side are correctly reconstructed, however.


I also forgot to mention that in the OP, you seem to have been affected by problems with transaction scanning combined with an issue that causes transactions to be stuck pending. This is a known issue but we are still tracking down the cause. Sometimes they get unstuck after some random period of time, so be cautious about resending transactions.

[bitmeat]

Great! Thanks for posting links to code!

Question:

Code Select Expand


new_entry = false;
+                     // TODO: We should probably really have a map of asset ids to amounts per ledger entry
+                     if( entry.amount.asset_id == amount.asset_id )
+                         entry.amount += amount;
+                     else if( entry.amount.amount == 0 )
+                         entry.amount = amount;


Will you check this logic. And also - are there tests associated with this code? I know it's a lot to ask for but we should have good integration tests that try all kinds of things.

E.g. have you thoroughly tested with multiple assets. Or able to simulate a missing transaction on the sending/receiving end, and see how the system behaves?

Thanks! Cheers! Good job!

[vikram]

I have been working on the transaction history and wallet rescanning code:

• https://github.com/BitShares/bitshares_toolkit/compare/bcd3d69772...7291385d48
• https://github.com/BitShares/bitshares_toolkit/compare/d43c3982a9...fc69f9efa1
• https://github.com/BitShares/bitshares_toolkit/compare/c09d3fecb1...f998c122ad
• https://github.com/BitShares/bitshares_toolkit/compare/23372db089...dc1d28ec71
• https://github.com/BitShares/bitshares_toolkit/commit/0e41da25a206518588e8b6deb5a5db3e0ad95b87

There were quite a number of problems with the wallet transaction scanning causing missing entries, incorrect balances, silently failing rescans, etc. In my testing so far, transaction history reconstruction works much better now. I am able to delete every transaction record (withdrawals, deposits, account operations, asset operations) from my wallet and it correctly brings them all back (sans any irrecoverable information due to TITAN). More testing is required but I think we might be able to fix your issues.

These fixes will be in the next release. You guys will have to do a full rescan (backup your wallet first, just in case) and we can see if it fixes your transaction histories.

[bytemaster]

Agreed.

[Simeon II]

Here is what is going on:

He has received all transactions, but when you "re-scan" you do not detect outgoing transactions. 

So when you cleared pending transactions you deleted your knowledge of those transactions from your wallet.   His transaction record is accurate, your balance is accurate, but your transaction record is missing those 2 transactions.

Are you saying that unless I keep my local hard drive safe, I have no way of ever reconstructing the missing transactions in the ledger?

I mean if I export my wallet, then some transactions occur, then my hard drive dies , will I be able to re-create my account from scratch from that older wallet?
(This happened to me by the way and I lost 400 PTS during donation period, because I thought Qt PTS used the same change address, so my backup didn't work). I also hope your change addresses are deterministic.

All private keys we generate are deterministic and can be regenerated.... as long as you have registered all of your accounts you will have an easy time recovering your balance, but your detailed account history could be gone for ever.  If you have unregistered accounts and do not know the public keys for them, then you will have to brute force scan with all of your keys and that could take an eternity (doable, but painful).   

For this reason I recommend registering all accounts on the blockchain *or* backing up all public keys for the unregistered accounts you do create.

This should be posted some place more prominent!

[bytemaster]

Here is what is going on:

He has received all transactions, but when you "re-scan" you do not detect outgoing transactions. 

So when you cleared pending transactions you deleted your knowledge of those transactions from your wallet.   His transaction record is accurate, your balance is accurate, but your transaction record is missing those 2 transactions.

Are you saying that unless I keep my local hard drive safe, I have no way of ever reconstructing the missing transactions in the ledger?

I mean if I export my wallet, then some transactions occur, then my hard drive dies , will I be able to re-create my account from scratch from that older wallet?
(This happened to me by the way and I lost 400 PTS during donation period, because I thought Qt PTS used the same change address, so my backup didn't work). I also hope your change addresses are deterministic.

All private keys we generate are deterministic and can be regenerated.... as long as you have registered all of your accounts you will have an easy time recovering your balance, but your detailed account history could be gone for ever.   If you have unregistered accounts and do not know the public keys for them, then you will have to brute force scan with all of your keys and that could take an eternity (doable, but painful).   

For this reason I recommend registering all accounts on the blockchain *or* backing up all public keys for the unregistered accounts you do create.

[toast]

Correct, told me that he *does* have the correct balance and is only missing the transaction for the 10k btsx

krabbypatty must have contacted you regarding this.

I just want to add that while my incoming transaction are about ~130K my combined accounts  balance is 140K -pretty much where it should be, except I have no way to tell who paid the 10K difference.

[bitmeat]

The transaction with no memo for 20K is missing on receive side... very interesting. 

Does his ledger balance out with his account balance after a rescan?

I've instructed him to do a rescan as well. I think he was able to send the balance, but never saw it in the ledger.

I think he is also missing the transaction with 10K one of the "2nd batch" transactions - there are 2 with the same memo, but he got only one - look at TX ID on the right

[bytemaster]

The transaction with no memo for 20K is missing on receive side... very interesting. 

Does his ledger balance out with his account balance after a rescan?