st/xorg: fix masked transformations