Colored coins and Mastercoins are essentially add on's to the bitcoin protocol, they function by using the bitcoin blockchain.
Colored coins are stock/bonds that people send/issue through their bitcoin wallets. They represent whatever the issuer decidedness
Mastercoins has a few features such as "savings wallets", and "betting" (futures markets kinda that allow you to hold value at the price of (insert anything).
Protoshares and bitshares are a different ball game. They don't sit on top of the bitcoin blockchain. Protoshares are basically stock in what Invictus innovations does. Any protoshares you own will be forked into the DACs invictus creates. One of these is bitshares, there is a lot going on with bitshares and you can learn about them on this forum, but to dumb it down to a sentence they are tackling some of the same issues as mastercoin but not exactly the same. Bitshares hasn't come out yet so protoshares are the only way to get them right now. Bitshares will pay dividends. If you think about bitshares like a distributed financial services corp that will pay profits back as dividends to bitshare owners.
Pays Dividends from Transaction Fees
Multiple Blockchains for Scalability and Different markets (commodities, stocks, currencies)
Does not rely upon trusted data feeds
Does not rely upon IOUs or bearer bonds
Colored Coins: Bearer Bonds layered on Bitcoin
Mastercoin: Colored Coins + BitShares like BitAssets implemented with Trusted (centralized) price feeds.
ProtoShares -> BitShares, DomainShares, LuckyShares, VoteShares, AP Shares
The primary problem with Mastercoin is that it will be limited in transaction volume by the Bitcoin network and they will eventually have to move to their own blockchain.
Mastercoin and Invictus are both well funded but going about it in different ways.