BitShares Forum

Main => Technical Support => Topic started by: svk on October 22, 2015, 05:26:31 pm

Title: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: svk on October 22, 2015, 05:26:31 pm
Thanks to awesome work by theoretical and jcalfee we now have two simple command line tools that will remove keys that are not in the Bitshares 2 genesis file from your keys json file generated by the wallet_export_keys command. I've just tested it on a wallet with 95k keys (16MB) and it was reduced to 375 keys (64kB). This might get embedded in the Bitshares 2 wallet and in a future 0.9.4 release of legacy Bitshares, but I wanted to share it with those of you who are struggling with importing your keys.

DISCLAIMER: try at your own risk, I'm not responsible for any loss of keys or balances, etc. Instructions are for *nix/OSX
DISCAIMER 2: Please backup and keep safe your original keys file first

The tools were written using Python and Nodejs so you'll need those installed.

Step 1: Generate a bloom filter file from Bitshares 2 genesis file
Code: [Select]
python create_bloom_filter.py -i ./genesis.json --o bloom.dat
Step 2: Filter your old wallet.json using the bloom filter
Code: [Select]
cat OLD_WALLET_NAME.json | node ./bloom_filter_wallet.js > NEW_WALLET_NAME.json
The script will run, telling you as it goes how many keys it's removed and has processed.

Step 3: Profit
Once that's done, you should have a MUCH smaller json file containing the keys you need to import in the Bitshares 2.0 wallet of your choice, light wallet, OpenLedger, localhost, etc.
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: jamesc on October 22, 2015, 05:37:46 pm
This is from Step 1: https://www.dropbox.com/s/jw0r603u1klm7s5/bloom.dat
The sha1 sum will print when you run step two... It should print:
Code: [Select]
bloom.dat (1048576 bytes) sha1 3cee441d8d28ab3b26aea149630fa2a96a91845c
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: jamesc on October 22, 2015, 05:43:52 pm
...backup your original key file as it contains all your keys.
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: roadscape on October 23, 2015, 05:12:53 am
Much needed, thanks!! +5%
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: unreadPostsSinceLastVisit on October 23, 2015, 11:58:38 pm
made it to step 2

Code: [Select]
merockstar@merockstar-HP-Pavilion-dv6-Notebook-PC:~/Desktop/key pruning$ cat mig6.json | node ./bloom_filter_wallet.js > pruned1.json

module.js:340
    throw err;
          ^
Error: Cannot find module '../src/common/hash'
  at Function.Module._resolveFilename (module.js:338:15)
  at Function.Module._load (module.js:280:25)
  at Module.require (module.js:364:17)
  at require (module.js:380:17)
  at Object.<anonymous> (/home/merockstar/Desktop/key pruning/bloom_filter_wallet.js:20:9)
  at Module._compile (module.js:456:26)
  at Object.Module._extensions..js (module.js:474:10)
  at Module.load (module.js:356:32)
  at Function.Module._load (module.js:312:12)
  at Function.Module.runMain (module.js:497:10)
  at startup (node.js:119:16)
  at node.js:902:3

coffee script is installed
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: jamesc on October 24, 2015, 01:31:27 am
made it to step 2

Code: [Select]
merockstar@merockstar-HP-Pavilion-dv6-Notebook-PC:~/Desktop/key pruning$ cat mig6.json | node ./bloom_filter_wallet.js > pruned1.json

module.js:340
    throw err;
          ^
Error: Cannot find module '../src/common/hash'
  at Function.Module._resolveFilename (module.js:338:15)
  at Function.Module._load (module.js:280:25)
  at Module.require (module.js:364:17)
  at require (module.js:380:17)
  at Object.<anonymous> (/home/merockstar/Desktop/key pruning/bloom_filter_wallet.js:20:9)
  at Module._compile (module.js:456:26)
  at Object.Module._extensions..js (module.js:474:10)
  at Module.load (module.js:356:32)
  at Function.Module._load (module.js:312:12)
  at Function.Module.runMain (module.js:497:10)
  at startup (node.js:119:16)
  at node.js:902:3

coffee script is installe

The instructions do not give you all the files you need. One way is to clone the repository. I have a sample maybe by for Monday so you can do this in the web wallet instead.
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: NewMine on October 24, 2015, 01:39:52 am
This is all you have to do? There is no way anyone will have a problem with this method. /s
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: rajarush on October 24, 2015, 01:46:32 am
Oh man this is so simple! Thank you for the good work!
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: unreadPostsSinceLastVisit on October 24, 2015, 01:51:38 am
made it to step 2

Code: [Select]
merockstar@merockstar-HP-Pavilion-dv6-Notebook-PC:~/Desktop/key pruning$ cat mig6.json | node ./bloom_filter_wallet.js > pruned1.json

module.js:340
    throw err;
          ^
Error: Cannot find module '../src/common/hash'
  at Function.Module._resolveFilename (module.js:338:15)
  at Function.Module._load (module.js:280:25)
  at Module.require (module.js:364:17)
  at require (module.js:380:17)
  at Object.<anonymous> (/home/merockstar/Desktop/key pruning/bloom_filter_wallet.js:20:9)
  at Module._compile (module.js:456:26)
  at Object.Module._extensions..js (module.js:474:10)
  at Module.load (module.js:356:32)
  at Function.Module._load (module.js:312:12)
  at Function.Module.runMain (module.js:497:10)
  at startup (node.js:119:16)
  at node.js:902:3

coffee script is installe

The instructions do not give you all the files you need. One way is to clone the repository. I have a sample maybe by for Monday so you can do this in the web wallet instead.

clone what repository?
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: svk on October 24, 2015, 07:09:33 am
made it to step 2

Code: [Select]
merockstar@merockstar-HP-Pavilion-dv6-Notebook-PC:~/Desktop/key pruning$ cat mig6.json | node ./bloom_filter_wallet.js > pruned1.json

module.js:340
    throw err;
          ^
Error: Cannot find module '../src/common/hash'
  at Function.Module._resolveFilename (module.js:338:15)
  at Function.Module._load (module.js:280:25)
  at Module.require (module.js:364:17)
  at require (module.js:380:17)
  at Object.<anonymous> (/home/merockstar/Desktop/key pruning/bloom_filter_wallet.js:20:9)
  at Module._compile (module.js:456:26)
  at Object.Module._extensions..js (module.js:474:10)
  at Module.load (module.js:356:32)
  at Function.Module._load (module.js:312:12)
  at Function.Module.runMain (module.js:497:10)
  at startup (node.js:119:16)
  at node.js:902:3

coffee script is installe

The instructions do not give you all the files you need. One way is to clone the repository. I have a sample maybe by for Monday so you can do this in the web wallet instead.

clone what repository?

He means the graphene-ui repo. It looks like I missed the fact that the second script depends on some files from the graphene-ui repo, my bad. I'll update the instructions in the OP
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: nomoreheroes7 on October 24, 2015, 11:57:52 pm
Instructions are for *nix/OSX

...can this be done on Windows? I have some basic CLI-knowledge, but not much beyond simply copy/pasting commands.
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: svk on October 25, 2015, 06:12:51 am
Instructions are for *nix/OSX

...can this be done on Windows? I have some basic CLI-knowledge, but not much beyond simply copy/pasting commands.
I think so, its just a bit more work to get git, python and node installed, but apart from that there's nothing stopping this from working on windows.
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: Chris4210 on October 25, 2015, 04:05:05 pm
Can someone please create a walk through guide here? I have absolutely now idea what you guys are talking about. Some help would be great! Thank you
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: unreadPostsSinceLastVisit on October 25, 2015, 05:30:12 pm
Hooray I got it to work and now I have a wallet with some 258 keys!
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: unreadPostsSinceLastVisit on October 25, 2015, 05:33:18 pm
Can someone please create a walk through guide here? I have absolutely now idea what you guys are talking about. Some help would be great! Thank you

this guide is for linux. i think it's different if you're using windows.
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: NewMine on October 25, 2015, 05:35:28 pm
Instructions are for *nix/OSX

...can this be done on Windows? I have some basic CLI-knowledge, but not much beyond simply copy/pasting commands.
I think so, its just a bit more work to get git, python and node installed, but apart from that there's nothing stopping this from working on windows.
I think he or she was looking for a guide. Something not in dev speak. I'm waiting too.  I am glad you guys are working this problem but what you are proposing is something like if I ask you for directions to the supermarket and you respond with actual gps coordinates. We all know what they mean, just not all of us have the means to interpret or implement them.
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: unreadPostsSinceLastVisit on October 25, 2015, 05:57:37 pm
I don't seem to need a password to do the wallet prune.

newmine and chris, do you guys want me to follow the instructions on your wallets for you?

edit: I should point out that I haven't successfully claimed the balances on from the keys on the new wallet i've created for myself. but I got the script to run, and the hash checked out. runnign the balance import now and its taking forever still with 278 keys
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: NewMine on October 25, 2015, 10:50:08 pm
I don't seem to need a password to do the wallet prune.

newmine and chris, do you guys want me to follow the instructions on your wallets for you?

edit: I should point out that I haven't successfully claimed the balances on from the keys on the new wallet i've created for myself. but I got the script to run, and the hash checked out. runnign the balance import now and its taking forever still with 278 keys

Sure. Here is my private key: 5Hfkfi85akfmHPL12890aiyeW1xX6aLs9gdS
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: unreadPostsSinceLastVisit on October 26, 2015, 12:12:37 am
I don't seem to need a password to do the wallet prune.

newmine and chris, do you guys want me to follow the instructions on your wallets for you?

edit: I should point out that I haven't successfully claimed the balances on from the keys on the new wallet i've created for myself. but I got the script to run, and the hash checked out. runnign the balance import now and its taking forever still with 278 keys

Sure. Here is my private key: 5Hfkfi85akfmHPL12890aiyeW1xX6aLs9gdS

i assumed the private keys would be encrypted.
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: NewMine on October 26, 2015, 05:04:45 am
I don't seem to need a password to do the wallet prune.

newmine and chris, do you guys want me to follow the instructions on your wallets for you?

edit: I should point out that I haven't successfully claimed the balances on from the keys on the new wallet i've created for myself. but I got the script to run, and the hash checked out. runnign the balance import now and its taking forever still with 278 keys

Sure. Here is my private key: 5Hfkfi85akfmHPL12890aiyeW1xX6aLs9gdS

i assumed the private keys would be encrypted.
Why?
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: roadscape on October 26, 2015, 06:34:24 am
The export format is just a "minimal" version of the (encrypted) wallet, which is why you don't need to have your wallet unlocked when you export the keys. The keys file is as safe as the locked wallet itself.. not that I'd recommend sharing it
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: Musewhale on October 26, 2015, 07:11:19 am
cool +5%
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: bitcrab on October 26, 2015, 09:11:17 am
Instructions are for *nix/OSX

...can this be done on Windows? I have some basic CLI-knowledge, but not much beyond simply copy/pasting commands.
I think so, its just a bit more work to get git, python and node installed, but apart from that there's nothing stopping this from working on windows.
on windows the point is the last step: how to execute the cat command on windows?
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: cube on October 26, 2015, 09:35:46 am
on windows the point is the last step: how to execute the cat command on windows?

type

Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: EstefanTT on October 26, 2015, 02:20:30 pm
I'm not sure if I wait a simpler solution or I dig into this one.

Anyone knows how much time it could take to have a wallet base simpler solution ?

Anyone could give me an estimation of the time it could takes to me to acheive it with the explanations above ? I've wrote a few lines of html, css and autohotkey but my code knowledge is not very deep. I'm on windows 10 / light client.

Thanks ;)
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: Chris4210 on October 26, 2015, 02:25:55 pm
push the poll:

https://bitsharestalk.org/index.php/topic,19432.0.html

Please vote here
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: bitcrab on October 26, 2015, 03:38:15 pm
on windows the point is the last step: how to execute the cat command on windows?

type

thanks, I failed under Ubuntu but succeeded under windows.
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: Riverhead on October 26, 2015, 05:52:19 pm

This process is very useful thank you.

kept 168 of 25103 keys
kept 61 of 10157 keys

Needless to say the import process worked much better with the reduced number of keys.
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: jamesc on October 26, 2015, 06:50:19 pm
I'm not sure if I wait a simpler solution or I dig into this one.

Anyone knows how much time it could take to have a wallet base simpler solution ?

Anyone could give me an estimation of the time it could takes to me to acheive it with the explanations above ? I've wrote a few lines of html, css and autohotkey but my code knowledge is not very deep. I'm on windows 10 / light client.

Thanks ;)

Today for notes and tomorrow to test it.  So tomorrow or the day after...
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: EstefanTT on October 26, 2015, 07:04:45 pm
That's great news !!!

+500% !
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: youlonghun on October 29, 2015, 12:52:20 pm
waiting for an easy way,it hard for me in windows。 :'( :'( :'(
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: jamesc on October 29, 2015, 08:43:22 pm
We are testing the web version .. If it works well I can merge it and it will make its way into the installers:

Web release for importing large wallets (testers needed)
https://bitsharestalk.org/index.php/topic,19557.msg251364.html#top
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: mdj on October 31, 2015, 11:33:16 am
This worked for me however I am not seeing any of my NOTES in the import.
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: svk on October 31, 2015, 12:02:19 pm
This worked for me however I am not seeing any of my NOTES in the import.
This guide is for BTS, you can use the procedure with MUSE as well but some things need to be changed.

First you need to use the MUSE genesis.json file. Secondly, there are some BTS specific variables in one of the scripts. Look for GPH and replace it with BTS, and also BTS should be replaced with MUSE.
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: mdj on October 31, 2015, 12:53:26 pm
This worked for me however I am not seeing any of my NOTES in the import.
This guide is for BTS, you can use the procedure with MUSE as well but some things need to be changed.

First you need to use the MUSE genesis.json file. Secondly, there are some BTS specific variables in one of the scripts. Look for GPH and replace it with BTS, and also BTS should be replaced with MUSE.

Thanks! Will give it a try later
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: unreadPostsSinceLastVisit on November 04, 2015, 08:31:22 pm
I have a question.

Is the process different for muse? or would the same resulting wallet be okay to use for muse?

if it should be okay, then I have a bigger problem, because when I attempted to import my pruned wallet into muse, it only say my ags/pts notes, and not the ones I bought from the exchange, and it didn't import any of my names.
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: svk on November 04, 2015, 09:05:03 pm
I have a question.

Is the process different for muse? or would the same resulting wallet be okay to use for muse?

if it should be okay, then I have a bigger problem, because when I attempted to import my pruned wallet into muse, it only say my ags/pts notes, and not the ones I bought from the exchange, and it didn't import any of my names.

If you read the three posts before yours you'll have the answer.
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: unreadPostsSinceLastVisit on November 04, 2015, 09:54:24 pm
I have a question.

Is the process different for muse? or would the same resulting wallet be okay to use for muse?

if it should be okay, then I have a bigger problem, because when I attempted to import my pruned wallet into muse, it only say my ags/pts notes, and not the ones I bought from the exchange, and it didn't import any of my names.

If you read the three posts before yours you'll have the answer.

Aww man. Sorry.

Going to go try that now. wish me luck.

edit: looks like it worked. about to test.
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: cass on November 05, 2015, 10:52:24 am
works like a charme .. at least for me .. thx for this script!
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: ripplexiaoshan on November 08, 2015, 12:51:40 pm
Is there is window version?
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: Chris4210 on November 08, 2015, 06:32:39 pm
Yes, the filter is integrated in the light wallet too.

Just download the standard Windows light client and upload your backup to integrate to 2.0 .
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: Bitshiz on January 12, 2017, 02:51:41 am
Any help with this error. I cant upload my .json to the web client because I have too many keys and it crashes every fucking time. been trying to access my BTS for over a year and finally tried this manual but am getting this error. ANyone know what this is? And how to fix? This happens after I run
Code: [Select]
OLD_WALLET_NAME.json | node ./bloom_filter_wallet.js > NEW_WALLET_NAME.json but with my wallet names. Doing this through windows 10 64 and I have downloaded everything this tutorial says.


Code: [Select]
module.js:471
    throw err;
    ^

Error: Cannot find module '../web/lib/common/hash'
    at Function.Module._resolveFilename (module.js:469:15)
    at Function.Module._load (module.js:417:25)
    at Module.require (module.js:497:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (C:\Users\skysc\graphene-ui\programs\bloom_filter_wallet.js:20:9)
    at Module._compile (module.js:570:32)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Module.runMain (module.js:604:10)
    at run (bootstrap_node.js:394:7)
    at startup (bootstrap_node.js:149:9)
    at bootstrap_node.js:509:3
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: svk on January 13, 2017, 07:48:50 am
Any help with this error. I cant upload my .json to the web client because I have too many keys and it crashes every fucking time. been trying to access my BTS for over a year and finally tried this manual but am getting this error. ANyone know what this is? And how to fix? This happens after I run
Code: [Select]
OLD_WALLET_NAME.json | node ./bloom_filter_wallet.js > NEW_WALLET_NAME.json but with my wallet names. Doing this through windows 10 64 and I have downloaded everything this tutorial says.


Code: [Select]
module.js:471
    throw err;
    ^

Error: Cannot find module '../web/lib/common/hash'
    at Function.Module._resolveFilename (module.js:469:15)
    at Function.Module._load (module.js:417:25)
    at Module.require (module.js:497:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (C:\Users\skysc\graphene-ui\programs\bloom_filter_wallet.js:20:9)
    at Module._compile (module.js:570:32)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Module.runMain (module.js:604:10)
    at run (bootstrap_node.js:394:7)
    at startup (bootstrap_node.js:149:9)
    at bootstrap_node.js:509:3

Like Chris mentioned you could try the light wallet as it includes the bloom filter directly, as does https://bitshares.org/wallet I believe.

The bloom filter program in the graphene-ui repo is probably out of sync with the location of files in the repo right now after I moved a lot of things around, I'll have a look and update it if necessary.
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: svk on January 13, 2017, 08:25:59 am
Any help with this error. I cant upload my .json to the web client because I have too many keys and it crashes every fucking time. been trying to access my BTS for over a year and finally tried this manual but am getting this error. ANyone know what this is? And how to fix? This happens after I run
Code: [Select]
OLD_WALLET_NAME.json | node ./bloom_filter_wallet.js > NEW_WALLET_NAME.json but with my wallet names. Doing this through windows 10 64 and I have downloaded everything this tutorial says.


Code: [Select]
module.js:471
    throw err;
    ^

Error: Cannot find module '../web/lib/common/hash'
    at Function.Module._resolveFilename (module.js:469:15)
    at Function.Module._load (module.js:417:25)
    at Module.require (module.js:497:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (C:\Users\skysc\graphene-ui\programs\bloom_filter_wallet.js:20:9)
    at Module._compile (module.js:570:32)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Module.runMain (module.js:604:10)
    at run (bootstrap_node.js:394:7)
    at startup (bootstrap_node.js:149:9)
    at bootstrap_node.js:509:3

Ok I had a look and it was indeed trying to load modules that no longer exist. I pushed an update and also included a bloom_bitshares.dat file that you can use if don't want to compile your own.

To run the filter now, just go to the "bloom_filter" folder, run "npm i" to get the dependencies, name your old wallet file "wallet.json" then  do "npm start". The output will be in a file called "filtered_wallet.json"

I didn't have an old wallet.json file to test it with so the npm start method might not work properly, if so just run it manually like the original instructions said.
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: Bitshiz on January 13, 2017, 05:13:53 pm
Any help with this error. I cant upload my .json to the web client because I have too many keys and it crashes every fucking time. been trying to access my BTS for over a year and finally tried this manual but am getting this error. ANyone know what this is? And how to fix? This happens after I run
Code: [Select]
OLD_WALLET_NAME.json | node ./bloom_filter_wallet.js > NEW_WALLET_NAME.json but with my wallet names. Doing this through windows 10 64 and I have downloaded everything this tutorial says.


Code: [Select]
module.js:471
    throw err;
    ^

Error: Cannot find module '../web/lib/common/hash'
    at Function.Module._resolveFilename (module.js:469:15)
    at Function.Module._load (module.js:417:25)
    at Module.require (module.js:497:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (C:\Users\skysc\graphene-ui\programs\bloom_filter_wallet.js:20:9)
    at Module._compile (module.js:570:32)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Module.runMain (module.js:604:10)
    at run (bootstrap_node.js:394:7)
    at startup (bootstrap_node.js:149:9)
    at bootstrap_node.js:509:3

Like Chris mentioned you could try the light wallet as it includes the bloom filter directly, as does https://bitshares.org/wallet I believe.

The bloom filter program in the graphene-ui repo is probably out of sync with the location of files in the repo right now after I moved a lot of things around, I'll have a look and update it if necessary.
The light wallet crashes every time I try to load it. I suspect it is because my account has so many keys as in BTSX I was trading every day, many trades a day. I also have about 35 accounts but only used probably 4 of them although I did send small amounts to about 10  others out of the 35. The wallet file I am trying to import only had 3 acounts attached and is 1,013,686 KB, so it is a large json file. But every time i try to load it into the light client, after about 10 minutes the entire screen goes white. I have smaller wallet files with more accounts because it was before I regenerated many keys and when I load those it will eventually list all my accounts in the light client but when I hit claim balances, after an hour or more the client freezes up too.
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: Bitshiz on January 13, 2017, 05:16:55 pm
Any help with this error. I cant upload my .json to the web client because I have too many keys and it crashes every fucking time. been trying to access my BTS for over a year and finally tried this manual but am getting this error. ANyone know what this is? And how to fix? This happens after I run
Code: [Select]
OLD_WALLET_NAME.json | node ./bloom_filter_wallet.js > NEW_WALLET_NAME.json but with my wallet names. Doing this through windows 10 64 and I have downloaded everything this tutorial says.


Code: [Select]
module.js:471
    throw err;
    ^

Error: Cannot find module '../web/lib/common/hash'
    at Function.Module._resolveFilename (module.js:469:15)
    at Function.Module._load (module.js:417:25)
    at Module.require (module.js:497:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (C:\Users\skysc\graphene-ui\programs\bloom_filter_wallet.js:20:9)
    at Module._compile (module.js:570:32)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Module.runMain (module.js:604:10)
    at run (bootstrap_node.js:394:7)
    at startup (bootstrap_node.js:149:9)
    at bootstrap_node.js:509:3

Ok I had a look and it was indeed trying to load modules that no longer exist. I pushed an update and also included a bloom_bitshares.dat file that you can use if don't want to compile your own.

To run the filter now, just go to the "bloom_filter" folder, run "npm i" to get the dependencies, name your old wallet file "wallet.json" then  do "npm start". The output will be in a file called "filtered_wallet.json"

I didn't have an old wallet.json file to test it with so the npm start method might not work properly, if so just run it manually like the original instructions said.
Thank you! I will try it today.
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: Bitshiz on January 13, 2017, 07:18:29 pm
Any help with this error. I cant upload my .json to the web client because I have too many keys and it crashes every fucking time. been trying to access my BTS for over a year and finally tried this manual but am getting this error. ANyone know what this is? And how to fix? This happens after I run
Code: [Select]
OLD_WALLET_NAME.json | node ./bloom_filter_wallet.js > NEW_WALLET_NAME.json but with my wallet names. Doing this through windows 10 64 and I have downloaded everything this tutorial says.


Code: [Select]
module.js:471
    throw err;
    ^

Error: Cannot find module '../web/lib/common/hash'
    at Function.Module._resolveFilename (module.js:469:15)
    at Function.Module._load (module.js:417:25)
    at Module.require (module.js:497:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (C:\Users\skysc\graphene-ui\programs\bloom_filter_wallet.js:20:9)
    at Module._compile (module.js:570:32)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Module.runMain (module.js:604:10)
    at run (bootstrap_node.js:394:7)
    at startup (bootstrap_node.js:149:9)
    at bootstrap_node.js:509:3

Ok I had a look and it was indeed trying to load modules that no longer exist. I pushed an update and also included a bloom_bitshares.dat file that you can use if don't want to compile your own.

To run the filter now, just go to the "bloom_filter" folder, run "npm i" to get the dependencies, name your old wallet file "wallet.json" then  do "npm start". The output will be in a file called "filtered_wallet.json"

I didn't have an old wallet.json file to test it with so the npm start method might not work properly, if so just run it manually like the original instructions said.

so I tried it and got this error because of the 'cat' command:
Code: [Select]
'cat' is not recognized as an internal or external command,
operable program or batch file.

npm ERR! Windows_NT 10.0.14393
npm ERR! argv "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "start"
npm ERR! node v6.9.4
npm ERR! npm  v3.10.10
npm ERR! code ELIFECYCLE
npm ERR! bitshares-keys-bloom-filter@1.0.0 start: `cat wallet.json | node bloom_filter_wallet.js > filtered_wallet.json`
npm ERR! Exit status 255
npm ERR!
npm ERR! Failed at the bitshares-keys-bloom-filter@1.0.0 start script 'cat wallet.json | node bloom_filter_wallet.js > filtered_wallet.json'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the bitshares-keys-bloom-filter package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     cat wallet.json | node bloom_filter_wallet.js > filtered_wallet.json
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs bitshares-keys-bloom-filter
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls bitshares-keys-bloom-filter
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     C:\Users\skysc\graphene-ui\bloom_filter\npm-debug.log

C:\Users\skysc\graphene-ui\bloom_filter>

I then tried to follow instructions from the original post and got error:

Code: [Select]
C:\Users\skysc\graphene-ui\bloom_filter>wallet.json | node ./bloom_filter_wallet.js > filtered_wallet.json'
C:\Users\skysc\graphene-ui\bloom_filter\bloom_filter_wallet.js:29
    if (err) throw err
             ^

Error: ENOENT: no such file or directory, open 'C:\Users\skysc\graphene-ui\bloom_filter\bloom.dat'
    at Error (native)


So then i decided to switch out my original bloom.dat file I created yesterday instaed of using the bloom_bitshares.dat and then got this error:

Code: [Select]
C:\Users\skysc\graphene-ui\bloom_filter>wallet.json | node ./bloom_filter_wallet.js > filtered_wallet.json
bloom_bitshares.dat (1048576 bytes) sha

undefined:1



SyntaxError: Unexpected end of JSON input
    at Object.parse (native)
    at Socket.<anonymous> (C:\Users\skysc\graphene-ui\bloom_filter\bloom_filter_wallet.js:59:27)
    at emitNone (events.js:91:20)
    at Socket.emit (events.js:185:7)
    at endReadableNT (_stream_readable.js:974:12)
    at _combinedTickCallback (internal/process/next_tick.js:74:11)
    at process._tickCallback (internal/process/next_tick.js:98:9)
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: svk on January 18, 2017, 10:28:59 am
Any help with this error. I cant upload my .json to the web client because I have too many keys and it crashes every fucking time. been trying to access my BTS for over a year and finally tried this manual but am getting this error. ANyone know what this is? And how to fix? This happens after I run
Code: [Select]
OLD_WALLET_NAME.json | node ./bloom_filter_wallet.js > NEW_WALLET_NAME.json but with my wallet names. Doing this through windows 10 64 and I have downloaded everything this tutorial says.


Code: [Select]
module.js:471
    throw err;
    ^

Error: Cannot find module '../web/lib/common/hash'
    at Function.Module._resolveFilename (module.js:469:15)
    at Function.Module._load (module.js:417:25)
    at Module.require (module.js:497:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (C:\Users\skysc\graphene-ui\programs\bloom_filter_wallet.js:20:9)
    at Module._compile (module.js:570:32)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Module.runMain (module.js:604:10)
    at run (bootstrap_node.js:394:7)
    at startup (bootstrap_node.js:149:9)
    at bootstrap_node.js:509:3

Ok I had a look and it was indeed trying to load modules that no longer exist. I pushed an update and also included a bloom_bitshares.dat file that you can use if don't want to compile your own.

To run the filter now, just go to the "bloom_filter" folder, run "npm i" to get the dependencies, name your old wallet file "wallet.json" then  do "npm start". The output will be in a file called "filtered_wallet.json"

I didn't have an old wallet.json file to test it with so the npm start method might not work properly, if so just run it manually like the original instructions said.

so I tried it and got this error because of the 'cat' command:
Code: [Select]
'cat' is not recognized as an internal or external command,
operable program or batch file.

npm ERR! Windows_NT 10.0.14393
npm ERR! argv "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "start"
npm ERR! node v6.9.4
npm ERR! npm  v3.10.10
npm ERR! code ELIFECYCLE
npm ERR! bitshares-keys-bloom-filter@1.0.0 start: `cat wallet.json | node bloom_filter_wallet.js > filtered_wallet.json`
npm ERR! Exit status 255
npm ERR!
npm ERR! Failed at the bitshares-keys-bloom-filter@1.0.0 start script 'cat wallet.json | node bloom_filter_wallet.js > filtered_wallet.json'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the bitshares-keys-bloom-filter package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     cat wallet.json | node bloom_filter_wallet.js > filtered_wallet.json
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs bitshares-keys-bloom-filter
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls bitshares-keys-bloom-filter
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     C:\Users\skysc\graphene-ui\bloom_filter\npm-debug.log

C:\Users\skysc\graphene-ui\bloom_filter>

I then tried to follow instructions from the original post and got error:

Code: [Select]
C:\Users\skysc\graphene-ui\bloom_filter>wallet.json | node ./bloom_filter_wallet.js > filtered_wallet.json'
C:\Users\skysc\graphene-ui\bloom_filter\bloom_filter_wallet.js:29
    if (err) throw err
             ^

Error: ENOENT: no such file or directory, open 'C:\Users\skysc\graphene-ui\bloom_filter\bloom.dat'
    at Error (native)


So then i decided to switch out my original bloom.dat file I created yesterday instaed of using the bloom_bitshares.dat and then got this error:

Code: [Select]
C:\Users\skysc\graphene-ui\bloom_filter>wallet.json | node ./bloom_filter_wallet.js > filtered_wallet.json
bloom_bitshares.dat (1048576 bytes) sha

undefined:1



SyntaxError: Unexpected end of JSON input
    at Object.parse (native)
    at Socket.<anonymous> (C:\Users\skysc\graphene-ui\bloom_filter\bloom_filter_wallet.js:59:27)
    at emitNone (events.js:91:20)
    at Socket.emit (events.js:185:7)
    at endReadableNT (_stream_readable.js:974:12)
    at _combinedTickCallback (internal/process/next_tick.js:74:11)
    at process._tickCallback (internal/process/next_tick.js:98:9)

Those instructions assume a Linux environment, the cat command is not available on Windows. You also need to fetch the latest version, there is no long a reference to web/lib/common/hash so apparently you're not using the latest release.

I've identified a bug however so I've pushed a new version just now, and this time I've tested it with an old wallet.json I had lying around so I know it works. I've also added a npm script for windows that you can use, try this in your node console:

Code: [Select]
npm run start-windows
Your wallet  json file must be called wallet.json for this to work.
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: Bitshiz on January 18, 2017, 05:16:55 pm
Well I just got this error now:
Quote
C:\Users\skysc\graphene-ui\bloom_filter>npm run start-windows

> bitshares-keys-bloom-filter@1.0.0 start-windows C:\Users\skysc\graphene-ui\bloom_filter
> type wallet.json | node bloom_filter_wallet.js > filtered_wallet.json

bloom_bitshares.dat (1048576 bytes) sha1 3cee441d8d28ab3b26aea149630fa2a96a91845c

C:\Users\skysc\graphene-ui\bloom_filter\bloom_filter_wallet.js:59
        var inputJSON = inputChunks.join("")
                                    ^

RangeError: Invalid string length
    at Array.join (native)
    at Socket.<anonymous> (C:\Users\skysc\graphene-ui\bloom_filter\bloom_filter_wallet.js:59:37)
    at emitNone (events.js:91:20)
    at Socket.emit (events.js:185:7)
    at endReadableNT (_stream_readable.js:974:12)
    at _combinedTickCallback (internal/process/next_tick.js:74:11)
    at process._tickCallback (internal/process/next_tick.js:98:9)

npm ERR! Windows_NT 10.0.14393
npm ERR! argv "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "run" "start-windows"
npm ERR! node v6.9.4
npm ERR! npm  v3.10.10
npm ERR! code ELIFECYCLE
npm ERR! bitshares-keys-bloom-filter@1.0.0 start-windows: `type wallet.json | node bloom_filter_wallet.js > filtered_wallet.json`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the bitshares-keys-bloom-filter@1.0.0 start-windows script 'type wallet.json | node bloom_filter_wallet.js > filtered_wallet.json'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the bitshares-keys-bloom-filter package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     type wallet.json | node bloom_filter_wallet.js > filtered_wallet.json
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs bitshares-keys-bloom-filter
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls bitshares-keys-bloom-filter
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     C:\Users\skysc\graphene-ui\bloom_filter\npm-debug.log



I deleted graphene-ui and cloned it again before I ran this. I then ran "npm install" and copied my wallet.json(I changed the name to wallet.json) file to the bloom_filter folder.

I figured it out! Kind of. My json file is over a 1,013,686KB. So it is too big of a json file. I tried on another wallet file that is only 90,252KB and it is working. Is there any way to get this to work on a larger file? I don't think I am missing any balances but just curious what the limit or why there would be a limit on the json file size?

Quote
kept 2099 of 534165 keys

kept 2102 of 534168 keys

kept 2103 of 534169 keys
Thanks @svk 
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: svk on January 18, 2017, 06:55:27 pm
Well I just got this error now:
Quote
C:\Users\skysc\graphene-ui\bloom_filter>npm run start-windows

> bitshares-keys-bloom-filter@1.0.0 start-windows C:\Users\skysc\graphene-ui\bloom_filter
> type wallet.json | node bloom_filter_wallet.js > filtered_wallet.json

bloom_bitshares.dat (1048576 bytes) sha1 3cee441d8d28ab3b26aea149630fa2a96a91845c

C:\Users\skysc\graphene-ui\bloom_filter\bloom_filter_wallet.js:59
        var inputJSON = inputChunks.join("")
                                    ^

RangeError: Invalid string length
    at Array.join (native)
    at Socket.<anonymous> (C:\Users\skysc\graphene-ui\bloom_filter\bloom_filter_wallet.js:59:37)
    at emitNone (events.js:91:20)
    at Socket.emit (events.js:185:7)
    at endReadableNT (_stream_readable.js:974:12)
    at _combinedTickCallback (internal/process/next_tick.js:74:11)
    at process._tickCallback (internal/process/next_tick.js:98:9)

npm ERR! Windows_NT 10.0.14393
npm ERR! argv "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "run" "start-windows"
npm ERR! node v6.9.4
npm ERR! npm  v3.10.10
npm ERR! code ELIFECYCLE
npm ERR! bitshares-keys-bloom-filter@1.0.0 start-windows: `type wallet.json | node bloom_filter_wallet.js > filtered_wallet.json`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the bitshares-keys-bloom-filter@1.0.0 start-windows script 'type wallet.json | node bloom_filter_wallet.js > filtered_wallet.json'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the bitshares-keys-bloom-filter package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     type wallet.json | node bloom_filter_wallet.js > filtered_wallet.json
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs bitshares-keys-bloom-filter
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls bitshares-keys-bloom-filter
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     C:\Users\skysc\graphene-ui\bloom_filter\npm-debug.log



I deleted graphene-ui and cloned it again before I ran this. I then ran "npm install" and copied my wallet.json(I changed the name to wallet.json) file to the bloom_filter folder.

I figured it out! Kind of. My json file is over a 1,013,686KB. So it is too big of a json file. I tried on another wallet file that is only 90,252KB and it is working. Is there any way to get this to work on a larger file? I don't think I am missing any balances but just curious what the limit or why there would be a limit on the json file size?

Quote
kept 2099 of 534165 keys

kept 2102 of 534168 keys

kept 2103 of 534169 keys
Thanks @svk

Wow that is a lot of keys! I have a 16MB json file myself but it only has around 85k keys.

If you're able to edit the file it's certainly possible to cut and paste the keys inside and assemble multiple smaller files. You'll have to do that by hand though, I recommend sublime editor for big JSON files like that.
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: Bitshiz on January 19, 2017, 02:02:43 am
Thanks!
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: Bitshiz on January 23, 2017, 05:45:10 pm
Instructions for the novice Windows user:

You will need to download nodejs, git, and Python 3 first and foremost.

Code: [Select]
cd C:\your\file\location\graphene-ui\bloom_filter
Code: [Select]
npm install
Code: [Select]
wallet.json
Code: [Select]
npm run start-windows
In the cmd window you will see it start to run and depending on how many keys you have it could take a little while. Also you should see the sha 1 sum at the beginning too (3cee441d8d28.......)
When it is finished you will have the a file with only the necessary keys linked to accounts and balances.

The new key file is in the graphene-ui/bloom_filter file and it is called
Code: [Select]
filtered_wallet.json
This just worked for me again to make sure my directions were correct. Unless someone changes things in the graphene-ui file you download from git.
Title: Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
Post by: svk on January 24, 2017, 11:52:43 am
Instructions for the novice Windows user:

You will need to download nodejs, git, and Python 3 first and foremost.

  • Open a command prompt by typing "cmd" in your windows search in the start menu
  • Clone the graphene-ui repoby typing "git clone https://github.com/cryptonomex/graphene-ui.git" in the cmd window. (remember that what ever directory/file location is showing on the prompt is where the graphen-ui download will be located
  • Now change the directory to the graphene-ui file location, type
Code: [Select]
cd C:\your\file\location\graphene-ui\bloom_filter
  • then type
Code: [Select]
npm install
  • Now find the wallet file you downloaded from Bitshares 0.93 and change the wallet name to
Code: [Select]
wallet.json
  • Copy the newly named wallet.json file
  • open the graphene-ui file location (the actual file, not in the cmd). Then open the "bloom_filter" file and paste the new wallet.json file inside
  • Go back to the command prompt and type
Code: [Select]
npm run start-windows

    In the cmd window you will see it start to run and depending on how many keys you have it could take a little while. Also you should see the sha 1 sum at the beginning too (3cee441d8d28.......)
    When it is finished you will have the a file with only the necessary keys linked to accounts and balances.

    The new key file is in the graphene-ui/bloom_filter file and it is called
    Code: [Select]
    filtered_wallet.json
    This just worked for me again to make sure my directions were correct. Unless someone changes things in the graphene-ui file you download from git.

    Thanks for the write-up!