util: return 0 for NaNs in float_to_ubyte