BitShares Forum

Main => General Discussion => Topic started by: bitmeat on July 20, 2014, 04:56:52 am

Title: [RESOLVED] Missing 30K - Help troubleshoot this.
Post by: bitmeat on July 20, 2014, 04:56:52 am
Ok so the story is as goes. I was trying to send 180K in chunks to another user. 2 of the transactions were showing as pending for me, and I cleared the pending transactions, rescanned/restarted/etc. so they disappeared, and I "had" my balance again.

So when I do from the console I get only the following transactions between me and lowfee-10prcnt (2 transactions missing one for 10K and one for 20K):

Code: [Select]
2014-07-20T02:31:59 8169      krabbypatty         lowfee-10prcnt      1.00000 BTSX            ensuring it works.                          0.10000 BTSX        a0da271
2014-07-20T02:37:40 8202      krabbypatty         lowfee-10prcnt      10,000.00000 BTSX       1st batch                                   0.10000 BTSX        41a6c3e
2014-07-20T02:54:15 8302      krabbypatty         lowfee-10prcnt      10,000.00000 BTSX       2nd batch                                   0.10000 BTSX        7e638b1
2014-07-20T03:28:21 8499      krabbypatty         lowfee-10prcnt      20,000.00000 BTSX       3rd batch                                   0.10000 BTSX        524dba1
2014-07-20T03:31:09 8515      krabbypatty         lowfee-10prcnt      20,000.00000 BTSX       4th batch                                   0.10000 BTSX        1af8cea
2014-07-20T03:57:48 8661      krabbypatty         lowfee-10prcnt      30,000.00000 BTSX       5th batch                                   0.10000 BTSX        baff8cc
2014-07-20T04:26:56 8830      krabbypatty         lowfee-10prcnt      20,000.00000 BTSX       6th batch                                   0.10000 BTSX        6eb27a0

On his side he has:

Code: [Select]
2014-07-20T02:33:17 8169      krabbypatty         lowfee-10prcnt      1.00000 BTSX            ensuring it works.                          0.00000 BTSX        a0da271
2014-07-20T02:37:35 8202      krabbypatty         lowfee-10prcnt      10,000.00000 BTSX       1st batch                                   0.00000 BTSX        41a6c3e
2014-07-20T02:53:10 8284      krabbypatty         lowfee-10prcnt      10,000.00000 BTSX       2nd batch                                   0.00000 BTSX        244f2da  <<<<<<<<<<<<<<<<<<
2014-07-20T02:53:10 8294      krabbypatty         lowfee-10prcnt      20,000.00000 BTSX                                                   0.00000 BTSX        8f4cea8  <<<<<<<<<<<<<<<<<<
2014-07-20T02:54:14 8302      krabbypatty         lowfee-10prcnt      10,000.00000 BTSX       2nd batch                                   0.00000 BTSX        7e638b1
2014-07-20T03:28:24 8499      krabbypatty         lowfee-10prcnt      20,000.00000 BTSX       3rd batch                                   0.00000 BTSX        524dba1
2014-07-20T03:31:04 8515      krabbypatty         lowfee-10prcnt      20,000.00000 BTSX       4th batch                                   0.00000 BTSX        1af8cea
2014-07-20T03:57:44 8661      krabbypatty         lowfee-10prcnt      30,000.00000 BTSX       5th batch                                   0.00000 BTSX        baff8cc
2014-07-20T04:26:55 8830      krabbypatty         lowfee-10prcnt      20,000.00000 BTSX       6th batch                                   0.00000 BTSX        6eb27a0

How do I reconcile my account to actually acknowledge that?

It says I have X amount, when I try to move X amount to another account it says insufficient funds and that I have X - 30K available BTSX. Clearly the system knows this happened, but it doesn't reflect it in my account.
Title: Re: Missing 30K - Help troubleshoot this.
Post by: bitmeat on July 20, 2014, 06:24:45 pm
Bump! Come on Invictus guys! I'm willing to work with you to solve this.

toast was the only one who even attempted looking at this. I would like an official word from someone who gets what's going on. I've provided logs and dumps to toast.

Title: Re: Missing 30K - Help troubleshoot this.
Post by: bytemaster on July 20, 2014, 06:25:56 pm
Hey there... you can imagine we all have our hands full right now, but we certainly want to help you solve it. 
Title: Re: Missing 30K - Help troubleshoot this.
Post by: bitmeat on July 20, 2014, 06:28:27 pm
Hey there... you can imagine we all have our hands full right now, but we certainly want to help you solve it.

Right. I'm actually mostly concerned that this could get a lot of people in trouble. Especially when this goes to exchanges. I think it should be somewhat higher priority.

Yes, I can imagine your hands are more than full, and it's Sunday. And you've done a great job. But I think this is more important than discussing how delegates work.
Title: Re: Missing 30K - Help troubleshoot this.
Post by: bytemaster on July 20, 2014, 06:29:43 pm
1) Did you put a memo in every transaction?   ie: can I assume those should be unique?
Title: Re: Missing 30K - Help troubleshoot this.
Post by: Riverhead on July 20, 2014, 06:31:07 pm



Have you done a wallet_account_balance in the console to see if it's just a UI bug?
Title: Re: Missing 30K - Help troubleshoot this.
Post by: bitmeat on July 20, 2014, 06:32:27 pm
1) Did you put a memo in every transaction?   ie: can I assume those should be unique?

I've sent the 2nd batch twice with the same memo, because I thought it didn't send, so I canceled pending and resent. However I had network issues at the time, I restarted the client, which by the way always crashes on Windows, and not sure if some vital piece didn't get written where it needs to. However the receiver has all the funds.

He is missing one transaction in the ledger, and I am missing 2 in mine. What I want to know is how does the number at the top of the UI get computer, where do I find it in the command line. And more importantly why is the ledger not reconstructed from the network on rescan?
Title: Re: Missing 30K - Help troubleshoot this.
Post by: bitmeat on July 20, 2014, 06:33:30 pm



Have you done a wallet_account_balance in the console to see if it's just a UI bug?

It's not just a UI bug, if you look at the post above there is a dump from wallet_account_transaction_history in the console.
Title: Re: Missing 30K - Help troubleshoot this.
Post by: bytemaster on July 20, 2014, 06:34:35 pm
Here is what is going on:

He has received all transactions, but when you "re-scan" you do not detect outgoing transactions. 

So when you cleared pending transactions you deleted your knowledge of those transactions from your wallet.   His transaction record is accurate, your balance is accurate, but your transaction record is missing those 2 transactions.
Title: Re: Missing 30K - Help troubleshoot this.
Post by: bitmeat on July 20, 2014, 06:38:05 pm
Here is what is going on:

He has received all transactions, but when you "re-scan" you do not detect outgoing transactions. 

So when you cleared pending transactions you deleted your knowledge of those transactions from your wallet.   His transaction record is accurate, your balance is accurate, but your transaction record is missing those 2 transactions.

Are you saying that unless I keep my local hard drive safe, I have no way of ever reconstructing the missing transactions in the ledger?

I mean if I export my wallet, then some transactions occur, then my hard drive dies , will I be able to re-create my account from scratch from that older wallet?
(This happened to me by the way and I lost 400 PTS during donation period, because I thought Qt PTS used the same change address, so my backup didn't work). I also hope your change addresses are deterministic.

Title: Re: Missing 30K - Help troubleshoot this.
Post by: Simeon II on July 20, 2014, 06:40:11 pm
Here is what is going on:

He has received all transactions, but when you "re-scan" you do not detect outgoing transactions. 

So when you cleared pending transactions you deleted your knowledge of those transactions from your wallet.   His transaction record is accurate, your balance is accurate, but your transaction record is missing those 2 transactions.

No it was not. Only the balance in the account (the total) was.
It is accurate now though, ~12 h later.
Title: Re: Missing 30K - Help troubleshoot this.
Post by: bytemaster on July 20, 2014, 06:40:15 pm
I think that we need to update the UI in the following way:

1) clear pending transactions should only hide the transactions (not remove them from the wallet)
2) When scanning the blockchain, we need to pick up transactions that are outgoing even if we lost information about who they were sent to... TITAN makes rescanning to restore history harder because it is just that private.

3) The balance shown for your account is based upon known spendable outputs and is the "true" balance after a rescan.   We are working to make sure the ledger accurately explains the true balance.

Title: Re: Missing 30K - Help troubleshoot this.
Post by: bitmeat on July 20, 2014, 06:48:06 pm
I think that we need to update the UI in the following way:

1) clear pending transactions should only hide the transactions (not remove them from the wallet)
2) When scanning the blockchain, we need to pick up transactions that are outgoing even if we lost information about who they were sent to... TITAN makes rescanning to restore history harder because it is just that private.

3) The balance shown for your account is based upon known spendable outputs and is the "true" balance after a rescan.   We are working to make sure the ledger accurately explains the true balance.

I see. Yes, even if it shows "lost encrypted transaction info" it's better than having discrepancies, which can be very confusing.

Also - and more concerning - there is a missing transaction in the ledger on the receiving side. I actually am not worried about my transactions missing, so long as he received them. I'm just trying to help here.

I'm thinking how would someone like BTER handle that? Will they just have receiving address, without an account name? Can they create unregistered receiving addresses - one per account, so that even if the tx info is lost, they still get the deposits?
Title: Re: Missing 30K - Help troubleshoot this.
Post by: bytemaster on July 20, 2014, 06:53:09 pm
There is a scalability issue with having many TITAN receive addresses because you must check every transaction against all keys and the cryptographic operation is expensive.  So BTER should have 1 account that they receive with.   

We have guidelines available for exchanges, but I need to find where the guys published it.   
Title: Re: Missing 30K - Help troubleshoot this.
Post by: bytemaster on July 20, 2014, 06:55:45 pm
The transaction with no memo for 20K is missing on receive side... very interesting. 

Does his ledger balance out with his account balance after a rescan?
Title: Re: Missing 30K - Help troubleshoot this.
Post by: bitmeat on July 20, 2014, 07:06:36 pm
The transaction with no memo for 20K is missing on receive side... very interesting. 

Does his ledger balance out with his account balance after a rescan?

I've instructed him to do a rescan as well. I think he was able to send the balance, but never saw it in the ledger.

I think he is also missing the transaction with 10K one of the "2nd batch" transactions - there are 2 with the same memo, but he got only one - look at TX ID on the right
Title: Re: Missing 30K - Help troubleshoot this.
Post by: toast on July 20, 2014, 07:08:18 pm
Correct, told me that he *does* have the correct balance and is only missing the transaction for the 10k btsx


Quote
krabbypatty must have contacted you regarding this.

 I just want to add that while my incoming transaction are about ~130K my combined accounts  balance is 140K -pretty much where it should be, except I have no way to tell who paid the 10K difference.
Title: Re: Missing 30K - Help troubleshoot this.
Post by: bytemaster on July 20, 2014, 07:14:37 pm
Here is what is going on:

He has received all transactions, but when you "re-scan" you do not detect outgoing transactions. 

So when you cleared pending transactions you deleted your knowledge of those transactions from your wallet.   His transaction record is accurate, your balance is accurate, but your transaction record is missing those 2 transactions.

Are you saying that unless I keep my local hard drive safe, I have no way of ever reconstructing the missing transactions in the ledger?

I mean if I export my wallet, then some transactions occur, then my hard drive dies , will I be able to re-create my account from scratch from that older wallet?
(This happened to me by the way and I lost 400 PTS during donation period, because I thought Qt PTS used the same change address, so my backup didn't work). I also hope your change addresses are deterministic.

All private keys we generate are deterministic and can be regenerated.... as long as you have registered all of your accounts you will have an easy time recovering your balance, but your detailed account history could be gone for ever.   If you have unregistered accounts and do not know the public keys for them, then you will have to brute force scan with all of your keys and that could take an eternity (doable, but painful).   

For this reason I recommend registering all accounts on the blockchain *or* backing up all public keys for the unregistered accounts you do create.
Title: Re: Missing 30K - Help troubleshoot this.
Post by: Simeon II on July 20, 2014, 07:20:52 pm
Here is what is going on:

He has received all transactions, but when you "re-scan" you do not detect outgoing transactions. 

So when you cleared pending transactions you deleted your knowledge of those transactions from your wallet.   His transaction record is accurate, your balance is accurate, but your transaction record is missing those 2 transactions.

Are you saying that unless I keep my local hard drive safe, I have no way of ever reconstructing the missing transactions in the ledger?

I mean if I export my wallet, then some transactions occur, then my hard drive dies , will I be able to re-create my account from scratch from that older wallet?
(This happened to me by the way and I lost 400 PTS during donation period, because I thought Qt PTS used the same change address, so my backup didn't work). I also hope your change addresses are deterministic.

All private keys we generate are deterministic and can be regenerated.... as long as you have registered all of your accounts you will have an easy time recovering your balance, but your detailed account history could be gone for ever.  If you have unregistered accounts and do not know the public keys for them, then you will have to brute force scan with all of your keys and that could take an eternity (doable, but painful).   

For this reason I recommend registering all accounts on the blockchain *or* backing up all public keys for the unregistered accounts you do create.

This should be posted some place more prominent!
Title: Re: Missing 30K - Help troubleshoot this.
Post by: bytemaster on July 20, 2014, 07:26:41 pm
Agreed.
Title: Re: Missing 30K - Help troubleshoot this.
Post by: vikram on July 23, 2014, 04:05:06 am
I have been working on the transaction history and wallet rescanning code:

There were quite a number of problems with the wallet transaction scanning causing missing entries, incorrect balances, silently failing rescans, etc. In my testing so far, transaction history reconstruction works much better now. I am able to delete every transaction record (withdrawals, deposits, account operations, asset operations) from my wallet and it correctly brings them all back (sans any irrecoverable information due to TITAN). More testing is required but I think we might be able to fix your issues.

These fixes will be in the next release. You guys will have to do a full rescan (backup your wallet first, just in case) and we can see if it fixes your transaction histories.
Title: Re: Missing 30K - Help troubleshoot this.
Post by: bitmeat on July 23, 2014, 04:24:36 am
Great! Thanks for posting links to code!

Question:

Code: [Select]
new_entry = false;
 +                     // TODO: We should probably really have a map of asset ids to amounts per ledger entry
 +                     if( entry.amount.asset_id == amount.asset_id )
 +                         entry.amount += amount;
 +                     else if( entry.amount.amount == 0 )
 +                         entry.amount = amount;

Will you check this logic. And also - are there tests associated with this code? I know it's a lot to ask for but we should have good integration tests that try all kinds of things.

E.g. have you thoroughly tested with multiple assets. Or able to simulate a missing transaction on the sending/receiving end, and see how the system behaves?

Thanks! Cheers! Good job!
Title: Re: Missing 30K - Help troubleshoot this.
Post by: vikram on July 23, 2014, 05:08:45 am
Great! Thanks for posting links to code!

Question:

Code: [Select]
new_entry = false;
 +                     // TODO: We should probably really have a map of asset ids to amounts per ledger entry
 +                     if( entry.amount.asset_id == amount.asset_id )
 +                         entry.amount += amount;
 +                     else if( entry.amount.amount == 0 )
 +                         entry.amount = amount;

Will you check this logic.

I see that this will fail when transferring assets other than the base asset, as I must have realized at some level before relegating the thought to the TODO there. Good catch! Will think how to address this.

Transactions can be almost arbitrarily complex but to keep meaningful records in our simple wallet we are making strong simplifying assumptions for now. We also have the problem that everyone already has a wallet using a certain format for storing their existing transaction records.

And also - are there tests associated with this code? I know it's a lot to ask for but we should have good integration tests that try all kinds of things.

E.g. have you thoroughly tested with multiple assets. Or able to simulate a missing transaction on the sending/receiving end, and see how the system behaves?

No tests yet, but I intend to write them when there is some time. Still many other fires to put out at the moment.

I have not thoroughly tested with multiple assets / market functionality yet, as is clear from the problem above. Missing transfers of the base asset on sending and receiving side are correctly reconstructed, however.


I also forgot to mention that in the OP, you seem to have been affected by problems with transaction scanning combined with an issue that causes transactions to be stuck pending. This is a known issue but we are still tracking down the cause. Sometimes they get unstuck after some random period of time, so be cautious about resending transactions.
Title: Re: Missing 30K - Help troubleshoot this.
Post by: liondani on July 23, 2014, 10:24:14 am
I have answerred that several times and it worked!

do this:

wallet_delegate_set_block_production ALL false

wallet_rescan_blockchain

wait 1-2 minutes (I think it does not matter )

wallet_delegate_set_block_production ALL true
Title: Re: Missing 30K - Help troubleshoot this.
Post by: xeroc on July 23, 2014, 11:29:33 am
Just a reminder for the reason of this:

TITAN transactions are costly and time consuming.. if you have a delegate running on the network you need to sign within 5-10 secs ... otherwise you miss your spot .. however if you have several 'other' accounts active TITAN scanning will probably not finish in time .. yet ... multithreading is on the way as BM told us somewhere :)
Title: Re: Missing 30K - Help troubleshoot this.
Post by: bitmeat on July 23, 2014, 04:14:43 pm
Liondani this didn't work for me. I still have discrepancies. This only fixes transactions that stay pending forever. But if I've lost information about pending transaction it's gone for good.
Title: Re: Missing 30K - Help troubleshoot this.
Post by: liondani on July 23, 2014, 04:48:34 pm
Liondani this didn't work for me. I still have discrepancies. This only fixes transactions that stay pending forever. But if I've lost information about pending transaction it's gone for good.

that must be a priority issue for the team I guess...
Title: Re: Missing 30K - Help troubleshoot this.
Post by: vikram on July 25, 2014, 08:53:15 pm
These fixes will be in the next release. You guys will have to do a full rescan (backup your wallet first, just in case) and we can see if it fixes your transaction histories.

Do you guys see any improvement using the latest version? https://github.com/dacsunlimited/bitsharesx/releases/tag/0.2.3
Title: Re: Missing 30K - Help troubleshoot this.
Post by: bitmeat on July 26, 2014, 12:42:57 am
These fixes will be in the next release. You guys will have to do a full rescan (backup your wallet first, just in case) and we can see if it fixes your transaction histories.

Do you guys see any improvement using the latest version? https://github.com/dacsunlimited/bitsharesx/releases/tag/0.2.3

Great! Using wallet_account_history and I now see a correct balance with the missing transactions from before shown as UNKNOWN.

However in the UI the balance column in the transaction history is now completely blank. as in not even numbers are seen there.

I'm closing this issue. I hope exchanges don't ever get to miss a transaction, that would be unfortunate.