Author Topic: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json  (Read 6675 times)

0 Members and 1 Guest are viewing this topic.

Offline svk

Thanks to awesome work by theoretical and jcalfee we now have two simple command line tools that will remove keys that are not in the Bitshares 2 genesis file from your keys json file generated by the wallet_export_keys command. I've just tested it on a wallet with 95k keys (16MB) and it was reduced to 375 keys (64kB). This might get embedded in the Bitshares 2 wallet and in a future 0.9.4 release of legacy Bitshares, but I wanted to share it with those of you who are struggling with importing your keys.

DISCLAIMER: try at your own risk, I'm not responsible for any loss of keys or balances, etc. Instructions are for *nix/OSX
DISCAIMER 2: Please backup and keep safe your original keys file first

The tools were written using Python and Nodejs so you'll need those installed.

Step 1: Generate a bloom filter file from Bitshares 2 genesis file
Code: [Select]
python create_bloom_filter.py -i ./genesis.json --o bloom.dat
Step 2: Filter your old wallet.json using the bloom filter
  • Clone the graphene-ui repo: "git clone https://github.com/cryptonomex/graphene-ui.git"
  • In a terminal, go inside the cloned repo: "cd graphene-ui/dl/" and install dependencies "npm install"
  • Copy the bloom.dat file you generated as well as your wallet.json file to "graphene-ui/dl/programs"
  • From inside the folder "graphene-ui/dl/programs", apply the filter to your wallet.json using:
Code: [Select]
cat OLD_WALLET_NAME.json | node ./bloom_filter_wallet.js > NEW_WALLET_NAME.json
The script will run, telling you as it goes how many keys it's removed and has processed.

Step 3: Profit
Once that's done, you should have a MUCH smaller json file containing the keys you need to import in the Bitshares 2.0 wallet of your choice, light wallet, OpenLedger, localhost, etc.
« Last Edit: October 24, 2015, 07:18:57 am by svk »
Worker: dev.bitsharesblocks

Offline jamesc

Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
« Reply #1 on: October 22, 2015, 05:37:46 pm »
This is from Step 1: https://www.dropbox.com/s/jw0r603u1klm7s5/bloom.dat
The sha1 sum will print when you run step two... It should print:
Code: [Select]
bloom.dat (1048576 bytes) sha1 3cee441d8d28ab3b26aea149630fa2a96a91845c

Offline jamesc

Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
« Reply #2 on: October 22, 2015, 05:43:52 pm »
...backup your original key file as it contains all your keys.
« Last Edit: October 22, 2015, 07:23:24 pm by jcalfee1 »

Offline roadscape

Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
« Reply #3 on: October 23, 2015, 05:12:53 am »
Much needed, thanks!! +5%
http://cryptofresh.com  |  witness: roadscape

Offline merockstar

Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
« Reply #4 on: October 23, 2015, 11:58:38 pm »
made it to step 2

Code: [Select]
[email protected]:~/Desktop/key pruning$ cat mig6.json | node ./bloom_filter_wallet.js > pruned1.json

module.js:340
    throw err;
          ^
Error: Cannot find module '../src/common/hash'
  at Function.Module._resolveFilename (module.js:338:15)
  at Function.Module._load (module.js:280:25)
  at Module.require (module.js:364:17)
  at require (module.js:380:17)
  at Object.<anonymous> (/home/merockstar/Desktop/key pruning/bloom_filter_wallet.js:20:9)
  at Module._compile (module.js:456:26)
  at Object.Module._extensions..js (module.js:474:10)
  at Module.load (module.js:356:32)
  at Function.Module._load (module.js:312:12)
  at Function.Module.runMain (module.js:497:10)
  at startup (node.js:119:16)
  at node.js:902:3

coffee script is installed

Offline jamesc

Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
« Reply #5 on: October 24, 2015, 01:31:27 am »
made it to step 2

Code: [Select]
[email protected]:~/Desktop/key pruning$ cat mig6.json | node ./bloom_filter_wallet.js > pruned1.json

module.js:340
    throw err;
          ^
Error: Cannot find module '../src/common/hash'
  at Function.Module._resolveFilename (module.js:338:15)
  at Function.Module._load (module.js:280:25)
  at Module.require (module.js:364:17)
  at require (module.js:380:17)
  at Object.<anonymous> (/home/merockstar/Desktop/key pruning/bloom_filter_wallet.js:20:9)
  at Module._compile (module.js:456:26)
  at Object.Module._extensions..js (module.js:474:10)
  at Module.load (module.js:356:32)
  at Function.Module._load (module.js:312:12)
  at Function.Module.runMain (module.js:497:10)
  at startup (node.js:119:16)
  at node.js:902:3

coffee script is installe

The instructions do not give you all the files you need. One way is to clone the repository. I have a sample maybe by for Monday so you can do this in the web wallet instead.

Offline NewMine

  • Hero Member
  • *****
  • Posts: 551
    • View Profile
Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
« Reply #6 on: October 24, 2015, 01:39:52 am »
This is all you have to do? There is no way anyone will have a problem with this method. /s

Offline rajarush

  • Jr. Member
  • **
  • Posts: 37
    • View Profile
Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
« Reply #7 on: October 24, 2015, 01:46:32 am »
Oh man this is so simple! Thank you for the good work!

Offline merockstar

Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
« Reply #8 on: October 24, 2015, 01:51:38 am »
made it to step 2

Code: [Select]
[email protected]:~/Desktop/key pruning$ cat mig6.json | node ./bloom_filter_wallet.js > pruned1.json

module.js:340
    throw err;
          ^
Error: Cannot find module '../src/common/hash'
  at Function.Module._resolveFilename (module.js:338:15)
  at Function.Module._load (module.js:280:25)
  at Module.require (module.js:364:17)
  at require (module.js:380:17)
  at Object.<anonymous> (/home/merockstar/Desktop/key pruning/bloom_filter_wallet.js:20:9)
  at Module._compile (module.js:456:26)
  at Object.Module._extensions..js (module.js:474:10)
  at Module.load (module.js:356:32)
  at Function.Module._load (module.js:312:12)
  at Function.Module.runMain (module.js:497:10)
  at startup (node.js:119:16)
  at node.js:902:3

coffee script is installe

The instructions do not give you all the files you need. One way is to clone the repository. I have a sample maybe by for Monday so you can do this in the web wallet instead.

clone what repository?

Offline svk

Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
« Reply #9 on: October 24, 2015, 07:09:33 am »
made it to step 2

Code: [Select]
[email protected]:~/Desktop/key pruning$ cat mig6.json | node ./bloom_filter_wallet.js > pruned1.json

module.js:340
    throw err;
          ^
Error: Cannot find module '../src/common/hash'
  at Function.Module._resolveFilename (module.js:338:15)
  at Function.Module._load (module.js:280:25)
  at Module.require (module.js:364:17)
  at require (module.js:380:17)
  at Object.<anonymous> (/home/merockstar/Desktop/key pruning/bloom_filter_wallet.js:20:9)
  at Module._compile (module.js:456:26)
  at Object.Module._extensions..js (module.js:474:10)
  at Module.load (module.js:356:32)
  at Function.Module._load (module.js:312:12)
  at Function.Module.runMain (module.js:497:10)
  at startup (node.js:119:16)
  at node.js:902:3

coffee script is installe

The instructions do not give you all the files you need. One way is to clone the repository. I have a sample maybe by for Monday so you can do this in the web wallet instead.

clone what repository?

He means the graphene-ui repo. It looks like I missed the fact that the second script depends on some files from the graphene-ui repo, my bad. I'll update the instructions in the OP
Worker: dev.bitsharesblocks

Offline nomoreheroes7

  • Hero Member
  • *****
  • Posts: 756
  • King of all the land
    • View Profile
  • BitShares: nomoreheroes7
Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
« Reply #10 on: October 24, 2015, 11:57:52 pm »
Instructions are for *nix/OSX

...can this be done on Windows? I have some basic CLI-knowledge, but not much beyond simply copy/pasting commands.
« Last Edit: October 25, 2015, 12:01:58 am by nomoreheroes7 »

Offline svk

Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
« Reply #11 on: October 25, 2015, 06:12:51 am »
Instructions are for *nix/OSX

...can this be done on Windows? I have some basic CLI-knowledge, but not much beyond simply copy/pasting commands.
I think so, its just a bit more work to get git, python and node installed, but apart from that there's nothing stopping this from working on windows.
Worker: dev.bitsharesblocks

Offline Chris4210

  • Sr. Member
  • ****
  • Posts: 431
  • Keep Building!
    • View Profile
    • www.payger.com
  • BitShares: chris4210
Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
« Reply #12 on: October 25, 2015, 04:05:05 pm »
Can someone please create a walk through guide here? I have absolutely now idea what you guys are talking about. Some help would be great! Thank you
Vote Chris4210 for Committee Member http://bit.ly/1WKC03B! | www.Payger.com - Payments + Messenger | www.BitShareshub.io - Community based fanpage for the BitShares Blockchain

Offline merockstar

Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
« Reply #13 on: October 25, 2015, 05:30:12 pm »
Hooray I got it to work and now I have a wallet with some 258 keys!

Offline merockstar

Re: GUIDE: How to remove unneeded keys from Bitshares 0.9.3 export_keys json
« Reply #14 on: October 25, 2015, 05:33:18 pm »
Can someone please create a walk through guide here? I have absolutely now idea what you guys are talking about. Some help would be great! Thank you

this guide is for linux. i think it's different if you're using windows.