ci/deqp-runner: Add a post-deqp-run filter list for known flakes.
So far, we've been putting our known flakes that intermittently fail CI
into the skips list. This has two downsides:
1) You don't know when the flakes stop happening and when to delist them
from skips, unless you go do a bunch of manual runs with the skips list
cleared.
2) If the flake was because the previous test left some broken state in
the HW, you may just move your intermittent to a new test.
With this new path, you can list your flakes in the flakes file to keep
them from erroring out people's pipelines. They still get run and
reported as is.
Reviewed-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6392>
diff --git a/.gitlab-ci/bare-metal/rootfs-setup.sh b/.gitlab-ci/bare-metal/rootfs-setup.sh
index 27a5c40..224c31f 100644
--- a/.gitlab-ci/bare-metal/rootfs-setup.sh
+++ b/.gitlab-ci/bare-metal/rootfs-setup.sh
@@ -31,6 +31,7 @@
DEQP_EXPECTED_FAILS \
DEQP_EXPECTED_RENDERER \
DEQP_NO_SAVE_RESULTS \
+ DEQP_FLAKES \
DEQP_PARALLEL \
DEQP_RUN_SUFFIX \
DEQP_SKIPS \
diff --git a/.gitlab-ci/deqp-runner.sh b/.gitlab-ci/deqp-runner.sh
index b96f272..c81229d 100755
--- a/.gitlab-ci/deqp-runner.sh
+++ b/.gitlab-ci/deqp-runner.sh
@@ -291,6 +291,27 @@
cp /tmp/deqp_runner.*.txt $RESULTS/
egrep -v ",Pass|,Skip|,ExpectedFail" $RESULTSFILE > $UNEXPECTED_RESULTSFILE
+ # deqp-runner's flake detection won't perfectly detect all flakes, so
+ # allow the driver to list some known flakes that won't intermittently
+ # fail people's pipelines (while still allowing them to run and be
+ # reported to IRC in the usual flake detection path). If we had some
+ # fails listed (so this wasn't a total runner failure), then filter out
+ # the known flakes and see if there are any issues left.
+ if [ -n "$DEQP_FLAKES" -a -s $UNEXPECTED_RESULTSFILE ]; then
+ set +x
+ while read line; do
+ line=`echo $line | sed 's|#.*||g'`
+ if [ -n "$line" ]; then
+ sed -i "/$line/d" $UNEXPECTED_RESULTSFILE
+ fi
+ done < $INSTALL/$DEQP_FLAKES
+ set -x
+
+ if [ ! -s $UNEXPECTED_RESULTSFILE ]; then
+ exit 0
+ fi
+ fi
+
if [ -z "$DEQP_NO_SAVE_RESULTS" ]; then
echo "Some unexpected results found (see cts-runner-results.txt in artifacts for full results):"
head -n 50 $UNEXPECTED_RESULTSFILE
diff --git a/.gitlab-ci/prepare-artifacts.sh b/.gitlab-ci/prepare-artifacts.sh
index e92063d..a6328ab 100755
--- a/.gitlab-ci/prepare-artifacts.sh
+++ b/.gitlab-ci/prepare-artifacts.sh
@@ -34,8 +34,7 @@
cp -Rp .gitlab-ci/fossils install/
cp -Rp .gitlab-ci/fossilize-runner.sh install/
cp -Rp .gitlab-ci/deqp-runner.sh install/
-cp -Rp .gitlab-ci/deqp-*-fails.txt install/
-cp -Rp .gitlab-ci/deqp-*-skips.txt install/
+cp -Rp .gitlab-ci/deqp-*.txt install/
# Tar up the install dir so that symlinks and hardlinks aren't each
# packed separately in the zip file.
