This is a great and much needed discussion. Many good considerations have been brought up.
Spectral's analysis aims at the level I think we should be thinking. Focusing on the witness pay is not the prime concern IMO, plus that can be adjusted as marketcap grows or as other factors dictate (supply of witnesses of adequate skill level and familiarity with BitShares technology). I say $300 / month is actually very low cost security. I think that is a good initial pay rate to start. Initially more attention to running witnesses will be required as the new code is set free into the wild. I think the first set of witnesses will be working pretty hard to keep up with code changes as bugs are discovered to sustain 99% reliability.
I also think a very important consideration is the voter apathy angle. For that reason I think the number of witnesses should not be too big to start off. IMO 30ish sounds like a good starting point until enough infrastructure (i.e. education, DPOSHUB, proxies, reliability tract records etc) is in place to get a clear picture of how to make adjustments.
There are a number of independent factors at play, like how well informed the biggest stakeholders are, how many proxies are active, what level of awareness the proxies have of what is actually going on in the network, what effect on perception bug discoveries might have, all these things could impact voting quality and quantity. Pretty damned hard to predict how effective voting will be with all that in play.
Who thought it would be so difficult to vote out delegates that aren't even producing blocks or have asked to be voted out in 1.0?
Who is John Galt?
I also think we need to be looking at the attack vector angle. Wackou & I are actively engaged at just that. The backbone infrastructure and "small world" approach can be an effective defense against attack, including DDoS attacks. Although it may be possible to manually "activate" a backup witness node, or switch a seed node to an active block producer within minutes from the start of an attack, I think it would be much better to automate such switching with quality monitoring. If backbone connections to witnesses can be monitored and they drop below a minimal threshold (which could be monitored by backbone nodes AND witnesses, i.e. both sides of the connection) a switch to alternate nodes can be done automatically. If witnesses can ascertain the health of the backbone nodes and it is good, they can limit their connections to backbone nodes exclusively, in which case the IP address of the witnesses is protected by the backbone nodes, and backbone nodes are likewise protected by seed nodes. If the health of the backbone drops below an acceptable threshold the witnesses turn to backups, alternates or even trusted seed nodes in a fallback cascade to insure network reliability. Witnesses may expose their IP address in such an event, but at least the network survives. It may buy enough time to get redundant backbone nodes up.
As for Riverhead's remarks, how does having alternate keys keep the original witness from coming back online and resuming block production with the original set of keys in addition to the new witness? Doesn't that require two separate witnesses that have to be independently voted in? What prevents 2 witness nodes operating under the same account name but using different keys? If only 1 is allowed to produce blocks which one chosen if 2 (or more) are configured as block producers?