+ create_lcssa_for_virtual_phi (loop);
+
+ /* If we're vectorizing an epilogue loop, the update_ssa above will
+ have ensured that the virtual operand is in SSA form throughout the
+ vectorized main loop. Normally it is possible to trace the updated
+ vector-stmt vdefs back to scalar-stmt vdefs and vector-stmt vuses
+ back to scalar-stmt vuses, meaning that the effect of the SSA update
+ remains local to the main loop. However, there are rare cases in
+ which the vectorized loop has vdefs even when the original scalar
+ loop didn't. For example, vectorizing a load with IFN_LOAD_LANES
+ introduces clobbers of the temporary vector array, which in turn
+ needs new vdefs. If the scalar loop doesn't write to memory, these
+ new vdefs will be the only ones in the vector loop.
+
+ In that case, update_ssa will have added a new virtual phi to the
+ main loop, which previously didn't need one. Ensure that we (locally)
+ maintain LCSSA form for the virtual operand, just as we would have
+ done if the virtual phi had existed from the outset. This makes it
+ easier to duplicate the scalar epilogue loop below. */
+ tree vop_to_rename = NULL_TREE;
+ if (loop_vec_info orig_loop_vinfo = LOOP_VINFO_ORIG_LOOP_INFO (loop_vinfo))
+ {
+ class loop *orig_loop = LOOP_VINFO_LOOP (orig_loop_vinfo);
+ vop_to_rename = create_lcssa_for_virtual_phi (orig_loop);
+ }
+