So the keys stay within my wallet folder. If I want to remove them, I can:
1. backup and rename the wallet folder
2. open the client, create a new wallet using the same passphare
3. if the money shows up, delete the renamed old wallet folder
Am I right?