[requesting help with] a 0.9.3c key file "compactor" for faster 2.0 imports

[xeroc]

+ owner keys never hold funds
+ active keys only hold funds if your account is registered as non-TITAN acount
+ otherwise all your funds are stored in individual balance-keys ..

you can get those keys by "wallet_account_balance_ids"

Here is a quick script I wrote that will give you the wif keys of your balanceids:
https://github.com/xeroc/bitshares-pytools/blob/master/tools/getbalancekeys.py

[BTSdac]

I want to build a tool that will strip a 0.9.3c export file of empty balance keys, as I understand this is the primary cause of slow imports.

The first thing I did was to pre-process the 2.0 genesis file and put each unique key on its own line.
There are 4 types of public keys: account-owner, account-active, balance, and vesting balance.

By my count, there is a total of 250,084 unique keys in the genesis file.

Here's a sample of the processed output:

Code: [Select]

BTS7w6nraWnLP2vvXhZJoiNw23ZzwZQpYe6EDQ86X4W4q9vRVSkKb pptall1 (active key) (lifetime)
BTS7w6nraWnLP2vvXhZJoiNw23ZzwZQpYe6EDQ86X4W4q9vRVSkKb pptall1 (owner key) (lifetime)
BTS5aR1DMjc1YLi82S1dUSxvUrxpuf52mKKEHF4H6EJr8YJxv6ZRk itunes (active key)
BTS5aR1DMjc1YLi82S1dUSxvUrxpuf52mKKEHF4H6EJr8YJxv6ZRk itunes (owner key)
BTS7N9L3mXEcyS5kBeYovZMjTAFVMNDY8e737jg2T6xa9x4YfKyDB qfhtw (active key) (lifetime)
BTS7N9L3mXEcyS5kBeYovZMjTAFVMNDY8e737jg2T6xa9x4YfKyDB qfhtw (owner key) (lifetime)
BTS8KeeWrZNMVksJFk4xW8EzMbfJxSG4WKMHKxeu9ghcQVenDdD3Y lfj (active key)
BTS8KeeWrZNMVksJFk4xW8EzMbfJxSG4WKMHKxeu9ghcQVenDdD3Y lfj (owner key)
BTS6sgieE6Hp1MHfX8xj3m4gQSai1ZYbwpSKLUuUyB145WQD9s1d1 asusa (active key) (lifetime)
BTS6sgieE6Hp1MHfX8xj3m4gQSai1ZYbwpSKLUuUyB145WQD9s1d1 asusa (owner key) (lifetime)
BTS84xPPonkZcssdD3CK4Ga6Uk878MEyybzAJnnVLBEvz67H18xRV lhb (active key)
BTS84xPPonkZcssdD3CK4Ga6Uk878MEyybzAJnnVLBEvz67H18xRV lhb (owner key)
BTS6AuyvQxZihktvkfY5Quazg27nu8uHRBedmUiPB9sn13M7ngDSV daniellarimer (active key)
BTS6AuyvQxZihktvkfY5Quazg27nu8uHRBedmUiPB9sn13M7ngDSV daniellarimer (owner key)
[....]
BTS1DabHAHdQeXFre6PC2YYGeayhuKuniVF 139895643 BTS
BTS1Dny3PirXmuRAMEWVi34fSsRWQpK7LZk 490645755 BTS
BTS1EFYgopx2r2zb9rzMAxtnfGn4GbPmpAa 21084 BTS
BTS1GFnzJf9q1QxegMufbHeWbRWD1mGXWdi 7070002 BTS
BTS1HMKhb6m9iHUKEUDTFdgxVSsgrsALwoQ 10000 BTS
BTS1JoujPuh25UY54dMqwSWVh46aEAE8gwi 4 CNY
BTS1Kkovbwo6eHxqxcrAxcMHotjUq5orGvQ 100000 BTS
BTS1KossAtqDrMc6NSX1EnCUbz9X5nnX93v 1 USD
BTS1Lc1y6bfwWzy2BvXTBLwEUyevK1Xh7Uh 1 CNY
BTS1M9qUwuJLt8ziqpkx4wgPaoWofQu1jmd 8459 BTS
[....]
BTSQEfUxEonuTvMzQ1Dju99SWVp6UisfUdQs 71656 BTS (vesting)
BTSQEiSzJERXqeCxbyy43sGdDr7fVRnQrWNx 6686334 BTS (vesting)
BTSQEigNwanoeCh7Gf4yErCaEAapG39KnJo2 9541 BTS (vesting)
BTSQEmYusBAvVFy6raYMYk51Pdd7GsaQzjpe 116821 BTS (vesting)
BTSQEo58CfiShWNWjAfyo5LE5uyBKgsQWgAq 321730 BTS (vesting)
BTSQEoQVa7LBaHLjSRRMaCZSRxmyV6U1iRh1 3892856327 BTS (vesting)
BTSQEpuL3mZR5jRSeYDvGb7iBZdXH42M2wCm 14952 BTS (vesting)
BTSQEq1uTzTb8hugonsGhugCo2T2vmq5QzdG 3574773 BTS (vesting)
BTSQEsL8W7oGQPExWDzc1EqDS6qKtVkKEZ1M 12998 BTS (vesting)
The next step is to parse the keys.json file and check what matches or doesn't.

I'm getting positive matches on account keys, but not balances.
So it seems the balance key format is different between the genesis and the export file.

Could someone explain the difference in formats and help me convert one to the other?

I also try to build a tool that any export key with balance ,
but I find all keys don`t have balance   

[xeroc]

Afaik devs came up with a blooming filter .. svk open a thread about this yesterday

[roadscape]

I want to build a tool that will strip a 0.9.3c export file of empty balance keys, as I understand this is the primary cause of slow imports.

The first thing I did was to pre-process the 2.0 genesis file and put each unique key on its own line.
There are 4 types of public keys: account-owner, account-active, balance, and vesting balance.

By my count, there is a total of 250,084 unique keys in the genesis file.

Here's a sample of the processed output:

Code: [Select]

BTS7w6nraWnLP2vvXhZJoiNw23ZzwZQpYe6EDQ86X4W4q9vRVSkKb pptall1 (active key) (lifetime)
BTS7w6nraWnLP2vvXhZJoiNw23ZzwZQpYe6EDQ86X4W4q9vRVSkKb pptall1 (owner key) (lifetime)
BTS5aR1DMjc1YLi82S1dUSxvUrxpuf52mKKEHF4H6EJr8YJxv6ZRk itunes (active key)
BTS5aR1DMjc1YLi82S1dUSxvUrxpuf52mKKEHF4H6EJr8YJxv6ZRk itunes (owner key)
BTS7N9L3mXEcyS5kBeYovZMjTAFVMNDY8e737jg2T6xa9x4YfKyDB qfhtw (active key) (lifetime)
BTS7N9L3mXEcyS5kBeYovZMjTAFVMNDY8e737jg2T6xa9x4YfKyDB qfhtw (owner key) (lifetime)
BTS8KeeWrZNMVksJFk4xW8EzMbfJxSG4WKMHKxeu9ghcQVenDdD3Y lfj (active key)
BTS8KeeWrZNMVksJFk4xW8EzMbfJxSG4WKMHKxeu9ghcQVenDdD3Y lfj (owner key)
BTS6sgieE6Hp1MHfX8xj3m4gQSai1ZYbwpSKLUuUyB145WQD9s1d1 asusa (active key) (lifetime)
BTS6sgieE6Hp1MHfX8xj3m4gQSai1ZYbwpSKLUuUyB145WQD9s1d1 asusa (owner key) (lifetime)
BTS84xPPonkZcssdD3CK4Ga6Uk878MEyybzAJnnVLBEvz67H18xRV lhb (active key)
BTS84xPPonkZcssdD3CK4Ga6Uk878MEyybzAJnnVLBEvz67H18xRV lhb (owner key)
BTS6AuyvQxZihktvkfY5Quazg27nu8uHRBedmUiPB9sn13M7ngDSV daniellarimer (active key)
BTS6AuyvQxZihktvkfY5Quazg27nu8uHRBedmUiPB9sn13M7ngDSV daniellarimer (owner key)
[....]
BTS1DabHAHdQeXFre6PC2YYGeayhuKuniVF 139895643 BTS
BTS1Dny3PirXmuRAMEWVi34fSsRWQpK7LZk 490645755 BTS
BTS1EFYgopx2r2zb9rzMAxtnfGn4GbPmpAa 21084 BTS
BTS1GFnzJf9q1QxegMufbHeWbRWD1mGXWdi 7070002 BTS
BTS1HMKhb6m9iHUKEUDTFdgxVSsgrsALwoQ 10000 BTS
BTS1JoujPuh25UY54dMqwSWVh46aEAE8gwi 4 CNY
BTS1Kkovbwo6eHxqxcrAxcMHotjUq5orGvQ 100000 BTS
BTS1KossAtqDrMc6NSX1EnCUbz9X5nnX93v 1 USD
BTS1Lc1y6bfwWzy2BvXTBLwEUyevK1Xh7Uh 1 CNY
BTS1M9qUwuJLt8ziqpkx4wgPaoWofQu1jmd 8459 BTS
[....]
BTSQEfUxEonuTvMzQ1Dju99SWVp6UisfUdQs 71656 BTS (vesting)
BTSQEiSzJERXqeCxbyy43sGdDr7fVRnQrWNx 6686334 BTS (vesting)
BTSQEigNwanoeCh7Gf4yErCaEAapG39KnJo2 9541 BTS (vesting)
BTSQEmYusBAvVFy6raYMYk51Pdd7GsaQzjpe 116821 BTS (vesting)
BTSQEo58CfiShWNWjAfyo5LE5uyBKgsQWgAq 321730 BTS (vesting)
BTSQEoQVa7LBaHLjSRRMaCZSRxmyV6U1iRh1 3892856327 BTS (vesting)
BTSQEpuL3mZR5jRSeYDvGb7iBZdXH42M2wCm 14952 BTS (vesting)
BTSQEq1uTzTb8hugonsGhugCo2T2vmq5QzdG 3574773 BTS (vesting)
BTSQEsL8W7oGQPExWDzc1EqDS6qKtVkKEZ1M 12998 BTS (vesting)
The next step is to parse the keys.json file and check what matches or doesn't.

I'm getting positive matches on account keys, but not balances.
So it seems the balance key format is different between the genesis and the export file.

Could someone explain the difference in formats and help me convert one to the other?