glsl: fix slot_end calculations and simplify reserved_slots check