panfrost: Assign primitive_size.pointer only if writes_point_size() returns true