I think you'd just have to crawl the chain and find all unspent outputs of transactions that voted for you. You could filter out recent transactions depending on your vote duration requirement, and the voters can verify their loyalty by signing with the same keys that signed the voting transactions.