ci/deqp-runner: Add a post-deqp-run filter list for known flakes.

So far, we've been putting our known flakes that intermittently fail CI
into the skips list.  This has two downsides:

1) You don't know when the flakes stop happening and when to delist them
   from skips, unless you go do a bunch of manual runs with the skips list
   cleared.

2) If the flake was because the previous test left some broken state in
   the HW, you may just move your intermittent to a new test.

With this new path, you can list your flakes in the flakes file to keep
them from erroring out people's pipelines.  They still get run and
reported as is.

Reviewed-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6392>

diff --git a/.gitlab-ci/bare-metal/rootfs-setup.sh b/.gitlab-ci/bare-metal/rootfs-setup.sh
index 27a5c40de2e568cf4cc9152a90bc77c3d504db85..224c31fc89ef93e8cae7b22e573f68e597d21fbd 100644 (file)
--- a/.gitlab-ci/bare-metal/rootfs-setup.sh
+++ b/.gitlab-ci/bare-metal/rootfs-setup.sh
@@ -31,6 +31,7 @@ for var in \
     DEQP_EXPECTED_FAILS \
     DEQP_EXPECTED_RENDERER \
     DEQP_NO_SAVE_RESULTS \
+    DEQP_FLAKES \
     DEQP_PARALLEL \
     DEQP_RUN_SUFFIX \
     DEQP_SKIPS \

diff --git a/.gitlab-ci/deqp-runner.sh b/.gitlab-ci/deqp-runner.sh
index b96f27208286e9e9a8ecc487fac617c89aa768d5..c81229d530da614b33ce3a16b23f7e0f75b5c4b5 100755 (executable)
--- a/.gitlab-ci/deqp-runner.sh
+++ b/.gitlab-ci/deqp-runner.sh
@@ -291,6 +291,27 @@ if [ $DEQP_EXITCODE -ne 0 ]; then
     cp /tmp/deqp_runner.*.txt $RESULTS/
     egrep -v ",Pass|,Skip|,ExpectedFail" $RESULTSFILE > $UNEXPECTED_RESULTSFILE
 
+    # deqp-runner's flake detection won't perfectly detect all flakes, so
+    # allow the driver to list some known flakes that won't intermittently
+    # fail people's pipelines (while still allowing them to run and be
+    # reported to IRC in the usual flake detection path).  If we had some
+    # fails listed (so this wasn't a total runner failure), then filter out
+    # the known flakes and see if there are any issues left.
+    if [ -n "$DEQP_FLAKES" -a -s $UNEXPECTED_RESULTSFILE ]; then
+        set +x
+        while read line; do
+            line=`echo $line | sed 's|#.*||g'`
+            if [ -n "$line" ]; then
+                sed -i "/$line/d" $UNEXPECTED_RESULTSFILE
+            fi
+        done < $INSTALL/$DEQP_FLAKES
+        set -x
+
+        if [ ! -s $UNEXPECTED_RESULTSFILE ]; then
+            exit 0
+        fi
+    fi
+
     if [ -z "$DEQP_NO_SAVE_RESULTS" ]; then
         echo "Some unexpected results found (see cts-runner-results.txt in artifacts for full results):"
         head -n 50 $UNEXPECTED_RESULTSFILE

diff --git a/.gitlab-ci/prepare-artifacts.sh b/.gitlab-ci/prepare-artifacts.sh
index e92063da94835307365a80cb73f3c4128af0375b..a6328ab5f80ec4fe20ec36540320ca24f88af70d 100755 (executable)
--- a/.gitlab-ci/prepare-artifacts.sh
+++ b/.gitlab-ci/prepare-artifacts.sh
@@ -34,8 +34,7 @@ cp -Rp .gitlab-ci/fossils.yml install/
 cp -Rp .gitlab-ci/fossils install/
 cp -Rp .gitlab-ci/fossilize-runner.sh install/
 cp -Rp .gitlab-ci/deqp-runner.sh install/
-cp -Rp .gitlab-ci/deqp-*-fails.txt install/
-cp -Rp .gitlab-ci/deqp-*-skips.txt install/
+cp -Rp .gitlab-ci/deqp-*.txt install/
 
 # Tar up the install dir so that symlinks and hardlinks aren't each
 # packed separately in the zip file.