Author Topic: What is the best way to download all of my account history?  (Read 3945 times)

0 Members and 1 Guest are viewing this topic.

Offline biophil

  • Hero Member
  • *****
  • Posts: 841
  • Incentives run the world
    • View Profile
    • Sign up for a Bitshares account!
  • BitShares: zebulon
What is the best way to download all of my account history?
« on: January 25, 2018, 08:11:16 pm »
I'm trying to compile a good record of my Bitshares activity, and the web wallet doesn't give you much history. Does anybody have a nice CSV history service or anything? Or a better web interface that lets you download history?

Offline oxarbitrage

Re: What is the best way to download all of my account history?
« Reply #1 on: January 25, 2018, 08:24:59 pm »
you can see all your(and others) full account history in the open explorer by going to the account page like http://open-explorer.io/#/accounts/biophil in the full account history section.
this uses a connection to a full traditional node.

then you can use also the csv exporter located here: http://open-explorer.io/bitshares-account-exporter/

and use your account id(1.2.1361), start date need to be something before 2015-10 and end date now to get all the history for your account.

the account exporter is not very friendly in its output right now but it will get all the operations made in a period of time, some improvements will came out soon to make it compatible with for example cointracking.info by exporting only the fill orders in a friendly format.

the exporter is connected to an elasticsearch bitshares node, this node is having some problems in the last few days as the server where it is located cant handle it anymore so it can be slow to export or down when you try. if this is the case let me know and ill send you the csv as i already downloaded it for your account now that it is working.

the elasticsearch node is going to be moved to a better server i already have in my power and the migration will be done between tomorrow and this weekend. this will allow to have a more stable elasticsearch node that will be able to bring richer features to the open-explorer and other applications.

hope it helps.

Offline biophil

  • Hero Member
  • *****
  • Posts: 841
  • Incentives run the world
    • View Profile
    • Sign up for a Bitshares account!
  • BitShares: zebulon
Re: What is the best way to download all of my account history?
« Reply #2 on: January 25, 2018, 09:12:36 pm »
Thanks for the pointer. I tried the CSV exporter, and it looks like something is wrong -- it hangs for a while (30 seconds?) and then gives me a CSV that only has column headers but no actual data. Nice explorer, by the way. I only just learned about it today; has it been operating a while?

I'll PM you my email address to send me what you exported!

Offline yvv

  • Hero Member
  • *****
  • Posts: 1182
    • View Profile
Re: What is the best way to download all of my account history?
« Reply #3 on: January 25, 2018, 11:27:58 pm »
It has 30 sec timeout hard coded. So you better retrieve your history in chunks, which can be processed in less than 30 sec.
 

Offline biophil

  • Hero Member
  • *****
  • Posts: 841
  • Incentives run the world
    • View Profile
    • Sign up for a Bitshares account!
  • BitShares: zebulon
Re: What is the best way to download all of my account history?
« Reply #4 on: January 26, 2018, 04:03:14 am »
It has 30 sec timeout hard coded. So you better retrieve your history in chunks, which can be processed in less than 30 sec.

I'll try that more deliberately, but I tried something like that already and didn't get anything. But I'll give it another go.

Offline oxarbitrage

Re: What is the best way to download all of my account history?
« Reply #5 on: January 26, 2018, 03:31:36 pm »
it's the server issue, i am configuring new server right now. will put the chain to sync in a bit and in some hours i will have you guys try the exporter again, everything should work fine.

Offline oxarbitrage

Re: What is the best way to download all of my account history?
« Reply #6 on: January 27, 2018, 02:39:52 pm »
Ok guys, sorry the delay. I had moved elasticsearch to a new node and now it is fully in sync and working. I changed the exporter to point to the new place, please try again:

http://open-explorer.io/bitshares-account-exporter/

I also increased the timeout to 60 seconds so big accounts can be downloaded without the need to go over it in chunks.

I confused at first with your account, thought it was biophil but i see now it is http://open-explorer.io/#/accounts/zebulon with 3782 operations.

I was able to download your account history now, sending the csv by email too but you can go ahead and try to download from the online tool.

thanks for the patience.

Offline Spady007

Re: What is the best way to download all of my account history?
« Reply #7 on: January 28, 2018, 05:57:56 am »
Hello. I'm also trying to download all of my transactions in order to reconcile my account. I followed the above steps and it appeared to work. The issue is the CSV file has plenty of data, but not the simple data I'm looking for - such as fill price and amount of bts purchased.

Appreciate any help!

Offline biophil

  • Hero Member
  • *****
  • Posts: 841
  • Incentives run the world
    • View Profile
    • Sign up for a Bitshares account!
  • BitShares: zebulon
Re: What is the best way to download all of my account history?
« Reply #8 on: January 30, 2018, 06:12:47 pm »
Hello. I'm also trying to download all of my transactions in order to reconcile my account. I followed the above steps and it appeared to work. The issue is the CSV file has plenty of data, but not the simple data I'm looking for - such as fill price and amount of bts purchased.

Appreciate any help!

I'll be writing some Python tools to parse the CSV file into human-readable data, if that's any help. I'll come back here with a github link when/if I get the tools working in a publicly accessible way.

Offline biophil

  • Hero Member
  • *****
  • Posts: 841
  • Incentives run the world
    • View Profile
    • Sign up for a Bitshares account!
  • BitShares: zebulon
Re: What is the best way to download all of my account history?
« Reply #9 on: January 30, 2018, 06:14:45 pm »
Ok guys, sorry the delay. I had moved elasticsearch to a new node and now it is fully in sync and working. I changed the exporter to point to the new place, please try again:

http://open-explorer.io/bitshares-account-exporter/

I also increased the timeout to 60 seconds so big accounts can be downloaded without the need to go over it in chunks.

I confused at first with your account, thought it was biophil but i see now it is http://open-explorer.io/#/accounts/zebulon with 3782 operations.

I was able to download your account history now, sending the csv by email too but you can go ahead and try to download from the online tool.

thanks for the patience.

Fantastic, looks like the tool works fine now, and I also got the CSV by email. Thanks very much!

Offline oxarbitrage

Re: What is the best way to download all of my account history?
« Reply #10 on: January 30, 2018, 06:19:09 pm »
Hello. I'm also trying to download all of my transactions in order to reconcile my account. I followed the above steps and it appeared to work. The issue is the CSV file has plenty of data, but not the simple data I'm looking for - such as fill price and amount of bts purchased.

Appreciate any help!

I'll be writing some Python tools to parse the CSV file into human-readable data, if that's any help. I'll come back here with a github link when/if I get the tools working in a publicly accessible way.

that will be awesome, i plan to improve it a bit but don't have the time so any help is very appreciated.

Offline Spady007

Re: What is the best way to download all of my account history?
« Reply #11 on: February 03, 2018, 05:17:05 am »
Looking forward to the update. Thanks a lot!

Offline Spady007

Re: What is the best way to download all of my account history?
« Reply #12 on: February 08, 2018, 12:21:29 am »
Just wondering if there has been some progress made on the transaction history?

Offline Spady007

Re: What is the best way to download all of my account history?
« Reply #13 on: February 21, 2018, 02:59:01 am »
Don't get me wrong, I'm a huge bitshares fan. However, getting a complete an accurate account history should be an automatic for an exchange!

Offline Spady007

Re: What is the best way to download all of my account history?
« Reply #14 on: February 24, 2018, 01:40:57 am »
OK.Ok.Ok

So I've spent 10 hours going over the csv script to figure out my transactions. Was making fabulous progress until I realized there appears to be no difference between a buy or sell order. So frustrating!! Can you tell the difference?? The top script is a sell from bts to bitUS. The bottom script is a buy from bitUS to bts.

Please help - I'm dying - just figuratively of course!

[4,{"fee":{"amount":0,"asset_id":"1.3.0"},"order_id":"1.7.49894451","account_id":"1.2.493907","pays":{"amount":66903,"asset_id":"1.3.121"},"receives":{"amount":1728763,"asset_id":"1.3.0"},"fill_price":{"base":{"amount":1935000,"asset_id":"1.3.121"},"quote":{"amount":50000000,"asset_id":"1.3.0"}},"is_maker":true}]
                                                                                 
[4,{"fee":{"amount":0,"asset_id":"1.3.0"},"order_id":"1.7.47254046","account_id":"1.2.493907","pays":{"amount":1126,"asset_id":"1.3.121"},"receives":{"amount":19263,"asset_id":"1.3.0"},"fill_price":{"base":{"amount":2750000,"asset_id":"1.3.121"},"quote":{"amount":47009350,"asset_id":"1.3.0"}},"is_maker":true}]