Fix a refcounting mistake with first_swap_fence.