The primary difference is that mining power cannot be taken away from those with the actual hardware and that you can purchase the hardware in secret.
Delegates have to have support from shareholders, are likely very public and trusted.
You only have to worry about a double spend when doing business with a delegate or someone in partnership with the delegate.