Ok, when I wrote OP I didn't pay attention to BTSX balance, but now it looks to me like the order got actually filled. I don't have any transaction for this fill in my transaction list, and I already rescanned the wallet a few times. I will wait for 0.4.24 before jumping to conclusions, but it seems to me that the fill might have happened at the same time as I hit the cancel button and it caused the glitch.