The logic is quite complicated and depends on several factors, most importantly it depends on the price feed. I don't think it's documented anywhere in detail (unless the source code counts).
Also see this issue https://github.com/BitShares/bitshares/issues/1354 .

I haven't seen any information as to why the listed bitUSD in a 200 USD short order at 180 bts/usd kept going down over time until this morning when it finally got filled and I only received 187 USD.  Set my interest rate at 1%. Why does this happen and where is it documented?