Keeping Quest Items After Theft:

If you steal a quest item, for example the Ahdarji's Heirloom (a ring of a countess), and get caught, you will lose all the stolen items except for the quest item because quest items cannot be taken out of your inventory except to further the quest.