From: Eric Anholt <eric@anholt.net>
Date: Wed, 19 Aug 2020 17:30:12 +0000 (-0700)
Subject: ci/deqp-runner: Add a post-deqp-run filter list for known flakes.
X-Git-Url: https://git.libre-soc.org/?a=commitdiff_plain;h=30da82c74c2175dd8ad09fd5d1a8aee8885d150a;p=mesa.git

ci/deqp-runner: Add a post-deqp-run filter list for known flakes.

So far, we've been putting our known flakes that intermittently fail CI
into the skips list.  This has two downsides:

1) You don't know when the flakes stop happening and when to delist them
   from skips, unless you go do a bunch of manual runs with the skips list
   cleared.

2) If the flake was because the previous test left some broken state in
   the HW, you may just move your intermittent to a new test.

With this new path, you can list your flakes in the flakes file to keep
them from erroring out people's pipelines.  They still get run and
reported as is.

Reviewed-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6392>
---

diff --git a/.gitlab-ci/bare-metal/rootfs-setup.sh b/.gitlab-ci/bare-metal/rootfs-setup.sh
index 27a5c40de2e..224c31fc89e 100644
--- a/.gitlab-ci/bare-metal/rootfs-setup.sh
+++ b/.gitlab-ci/bare-metal/rootfs-setup.sh
@@ -31,6 +31,7 @@ for var in \
     DEQP_EXPECTED_FAILS \
     DEQP_EXPECTED_RENDERER \
     DEQP_NO_SAVE_RESULTS \
+    DEQP_FLAKES \
     DEQP_PARALLEL \
     DEQP_RUN_SUFFIX \
     DEQP_SKIPS \
diff --git a/.gitlab-ci/deqp-runner.sh b/.gitlab-ci/deqp-runner.sh
index b96f2720828..c81229d530d 100755
--- a/.gitlab-ci/deqp-runner.sh
+++ b/.gitlab-ci/deqp-runner.sh
@@ -291,6 +291,27 @@ if [ $DEQP_EXITCODE -ne 0 ]; then
     cp /tmp/deqp_runner.*.txt $RESULTS/
     egrep -v ",Pass|,Skip|,ExpectedFail" $RESULTSFILE > $UNEXPECTED_RESULTSFILE
 
+    # deqp-runner's flake detection won't perfectly detect all flakes, so
+    # allow the driver to list some known flakes that won't intermittently
+    # fail people's pipelines (while still allowing them to run and be
+    # reported to IRC in the usual flake detection path).  If we had some
+    # fails listed (so this wasn't a total runner failure), then filter out
+    # the known flakes and see if there are any issues left.
+    if [ -n "$DEQP_FLAKES" -a -s $UNEXPECTED_RESULTSFILE ]; then
+        set +x
+        while read line; do
+            line=`echo $line | sed 's|#.*||g'`
+            if [ -n "$line" ]; then
+                sed -i "/$line/d" $UNEXPECTED_RESULTSFILE
+            fi
+        done < $INSTALL/$DEQP_FLAKES
+        set -x
+
+        if [ ! -s $UNEXPECTED_RESULTSFILE ]; then
+            exit 0
+        fi
+    fi
+
     if [ -z "$DEQP_NO_SAVE_RESULTS" ]; then
         echo "Some unexpected results found (see cts-runner-results.txt in artifacts for full results):"
         head -n 50 $UNEXPECTED_RESULTSFILE
diff --git a/.gitlab-ci/prepare-artifacts.sh b/.gitlab-ci/prepare-artifacts.sh
index e92063da948..a6328ab5f80 100755
--- a/.gitlab-ci/prepare-artifacts.sh
+++ b/.gitlab-ci/prepare-artifacts.sh
@@ -34,8 +34,7 @@ cp -Rp .gitlab-ci/fossils.yml install/
 cp -Rp .gitlab-ci/fossils install/
 cp -Rp .gitlab-ci/fossilize-runner.sh install/
 cp -Rp .gitlab-ci/deqp-runner.sh install/
-cp -Rp .gitlab-ci/deqp-*-fails.txt install/
-cp -Rp .gitlab-ci/deqp-*-skips.txt install/
+cp -Rp .gitlab-ci/deqp-*.txt install/
 
 # Tar up the install dir so that symlinks and hardlinks aren't each
 # packed separately in the zip file.