I think in addition to the (relatively small) fixed registration fee for witnesses, a witness should be required to deposit funds (of some fixed amount specified by the network) into a surety bond for them to be considered a valid candidate. These funds can be withdraw if the witness decides to retire (perhaps temporarily) but there would be a 2-week delay in that process (the retirement, meaning no more block production, would be immediate or at least would occur by the next maintenance perod, but the fund withdrawal would be delayed). A registered witness without a bond is not a valid candidate witness. People can still vote for those witnesses, but no matter how high their approval gets they will never become an active witness if they don't have a sufficient bond posted.
In addition to voting for a witness, stakeholders should optionally be able to vote to ban a particular witness. If the amount of stake voting to ban a particular witness exceeds some threshold, e.g. the median approval votes for active witnesses, the blockchain will ban that witness. Banning means that the blockchain takes away the funds in the surety bond from the witness and prevents that witness account from ever becoming a candidate witness again (obviously the person behind that account can always register a new witness account).
With this change there would be a financial incentive to not misbehave (for example by double signing blocks) even for witnesses who don't care about ever being a block producer again. It would also provide a mechanism for witnesses to legitimately retire without having to ask stakeholders to vote them out or compromise block production.
A more advanced alternative to banning witnesses would be to allow anyone to provide proof of a double sign and have the network do the banning automatically if a valid proof was provided (and also reward the proof provider from some fraction of the surety bond). A hybrid approach might also be desirable. For example, if the ban votes for a witness exceed some low threshold, then a double sign proof is enough to get the witness banned. If the ban votes exceed some higher threshold, the witness is banned without any crypographic proof required. This modification requires ensuring that the block signing protocol is designed in a way that small double signing proofs can be submitted and validated (I'm not sure if the current protocol is designed to support that).