panfrost/decode: Respect primitive size pointers