anv/descriptor_set: Don't fully destroy sets in pool destroy/reset