Run fewer graphics tests in blocking part of CQ/BVT.
TLDR: This reduces the number of graphics tests in BVT from 20 to 8
As autotest currently has a server overhead per client test of 90+
DUT holding seconds (crbug.com/726481), this saves more than
90s * (20-8) = 18 minutes in the locking suite (usually distributed
over 4-7 DUTs; or still 3-4 minutes CQ cycle time).
--
To make the Chrome OS commit queue work better we want to
a) reduce builder cycle time
b) reduce the number of possible flakes causing extra cycles
We also want to make the Chrome gardener's life easier and impose less
graphics tests to watch on the gardening rotation.
Notice all tests in this change are very mature and well behaved, but
the law of large numbers is against CQ runs. So the less valuable tests
must go out of the critical loop and into a not-blocking suite.
To not reduce the coverage frequency we move remaining tests to
bvt-perbuild. For this to work the graphics team will need to watch
regressions in a rotation and revert changes after failures are detected.
We move several very important tests to bvt-inline, which right now
finishes faster than bvt-cq and has greater hardware coverage.
The currently important/blocking graphics tests are
- graphics_GLBench.bvt - Chrome-like functionality, pixel accurate.
- graphics_GLMark2.bvt - Exercises game like paths.
- graphics_dEQP.bvt - gles2, gles3, gles31 and vulkan.
- graphics_Drm.bvt - Functional/integration tests (not in this
change will follow later).
- graphics_Gbm - Verifies Mesa graphics buffer management.
- graphics_Idle - Verifies graphics reaches low power state.
- graphics_Idle.arc - Moved by Richard to bvt-arc (good place).
- graphics_Sanity - Checks screenshot for corruption.
Sum of remaining pure test times (without autotest overhead) on
veyron_minnie R60-9574 are on the order of 3-4 minutes:
bin/autotest tests/graphics_GLBench/control.bvt - real 0m45.225s
bin/autotest tests/graphics_GLMark2/control.bvt - real 0m18.270s
bin/autotest tests/graphics_dEQP/control.bvt - real 0m11.459s
bin/autotest tests/graphics_Drm/control.bvt - real 0m55.893s
bin/autotest tests/graphics_Gbm/control - real 0m12.223s
While we are at it also get rid of JOB_RETRIES in bvt-perbuild.
BUG=chromium:723898, chromium:722474
TEST=CQ will test.
CQ-DEPEND=CL:*390814
Change-Id: I41e33557e74757c579ff0e9c5b0d65c0e60d0e6d
Reviewed-on: https://chromium-review.googlesource.com/510112
Commit-Ready: Ilja H. Friedel <ihf@chromium.org>
Tested-by: Ilja H. Friedel <ihf@chromium.org>
Reviewed-by: Stéphane Marchesin <marcheu@chromium.org>
diff --git a/client/site_tests/graphics_GLAPICheck/control b/client/site_tests/graphics_GLAPICheck/control
index 71f46ef..5154fc7 100644
--- a/client/site_tests/graphics_GLAPICheck/control
+++ b/client/site_tests/graphics_GLAPICheck/control
@@ -14,12 +14,11 @@
- If GL extensions don't include needed extensions
- If X extensions don't include DAMAGE and Composite
"""
-ATTRIBUTES = "suite:bvt-cq, suite:graphics, suite:graphics_per-day, suite:graphics_system, suite:hwqual"
+ATTRIBUTES = "suite:bvt-perbuild, suite:graphics, suite:graphics_per-day, suite:graphics_system, suite:hwqual"
TIME='SHORT'
TEST_CATEGORY = 'Performance'
TEST_CLASS = "graphics"
TEST_TYPE = 'client'
-JOB_RETRIES = 2
BUG_TEMPLATE = {
'components': ['OS>Kernel>Graphics'],
}
diff --git a/client/site_tests/graphics_GLBench/control.bvt-cq b/client/site_tests/graphics_GLBench/control.bvt
similarity index 82%
rename from client/site_tests/graphics_GLBench/control.bvt-cq
rename to client/site_tests/graphics_GLBench/control.bvt
index 2cbf161..58d6e1c 100644
--- a/client/site_tests/graphics_GLBench/control.bvt-cq
+++ b/client/site_tests/graphics_GLBench/control.bvt
@@ -2,11 +2,11 @@
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
-NAME = 'graphics_GLBench.bvt-cq'
+NAME = 'graphics_GLBench.bvt'
AUTHOR = 'chromeos-gfx'
PURPOSE = 'Benchmark the graphics library performance.'
CRITERIA = """
-On bvt-cq this test is not run as a benchmark. (One can find approximate
+On bvt this test is not run as a benchmark. (One can find approximate
performance numbers in the logs though, but they are ignored.) It still
generates output images and verifies their bit accuracy using MD5 checksums
which are stored in
@@ -20,7 +20,9 @@
Note: it should nearly never be required to remove old versions of good/bad
images from these directories.
"""
-ATTRIBUTES = "suite:bvt-cq, suite:graphics, suite:graphics_per-day, suite:graphics_system"
+# It is intentional that this suite is part of bvt-inline and bvt-perbuild
+# suites to get more stability data. Do not change this.
+ATTRIBUTES = "suite:bvt-inline, suite:bvt-perbuild, suite:graphics, suite:graphics_per-day, suite:graphics_system"
TIME='FAST'
TEST_CATEGORY = 'Performance'
TEST_CLASS = "gl"
@@ -31,7 +33,7 @@
}
DOC = """
-On bvt, bvt-cq we do not upload performance numbers to the chrome dashboard. We run
+On bvt we do not upload performance numbers to the chrome dashboard. We run
glbench with the "-hasty" option to stay well below the BVT limit of 20
minutes. This option will run each test at 512x512 resolution only, run it
only for a fraction of the loops we normally do, and not cool down the
diff --git a/client/site_tests/graphics_GLMark2/control.bvt-cq b/client/site_tests/graphics_GLMark2/control.bvt
similarity index 71%
rename from client/site_tests/graphics_GLMark2/control.bvt-cq
rename to client/site_tests/graphics_GLMark2/control.bvt
index ed51c2d..685f893 100644
--- a/client/site_tests/graphics_GLMark2/control.bvt-cq
+++ b/client/site_tests/graphics_GLMark2/control.bvt
@@ -2,10 +2,12 @@
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
-NAME = 'graphics_GLMark2.bvt-cq'
+NAME = 'graphics_GLMark2.bvt'
AUTHOR = 'chromeos-gfx'
PURPOSE = 'Benchmark the graphics library performance.'
-ATTRIBUTES = 'suite:bvt-cq'
+# It is intentional that this suite is part of bvt-inline and bvt-perbuild
+# suites to get more stability data. Do not change this.
+ATTRIBUTES = 'suite:bvt-inline, suite:bvt-perbuild, suite:graphics, suite:graphics_per-day, suite:graphics_system'
TIME='MEDIUM'
TEST_CATEGORY = 'Performance'
TEST_CLASS = "gl"
diff --git a/client/site_tests/graphics_Gbm/control b/client/site_tests/graphics_Gbm/control
index 6f32336..bf39193 100644
--- a/client/site_tests/graphics_Gbm/control
+++ b/client/site_tests/graphics_Gbm/control
@@ -8,7 +8,7 @@
CRITERIA = """
Tests the Mesa graphics buffer management. All tests must pass.
"""
-ATTRIBUTES = 'suite:graphics, suite:graphics_per-day, suite:bvt-cq, suite:graphics_system'
+ATTRIBUTES = 'suite:bvt-inline, suite:graphics, suite:graphics_per-day, suite:graphics_system'
TIME='FAST'
TEST_CATEGORY = 'Functional'
TEST_CLASS = "gl"
diff --git a/client/site_tests/graphics_GpuReset/control b/client/site_tests/graphics_GpuReset/control
index 27f336a..e2fb94c 100644
--- a/client/site_tests/graphics_GpuReset/control
+++ b/client/site_tests/graphics_GpuReset/control
@@ -13,7 +13,6 @@
TEST_CATEGORY = 'Functional'
TEST_CLASS = "gl"
TEST_TYPE = 'client'
-JOB_RETRIES = 2
# Reboot in the lab after the test ends.
DEPENDENCIES='cleanup-reboot'
BUG_TEMPLATE = {
diff --git a/client/site_tests/graphics_Idle/control b/client/site_tests/graphics_Idle/control
index d53e90a..c14272a 100644
--- a/client/site_tests/graphics_Idle/control
+++ b/client/site_tests/graphics_Idle/control
@@ -4,7 +4,7 @@
AUTHOR = 'chromeos-gfx'
NAME = "graphics_Idle"
-ATTRIBUTES = "suite:bvt-cq, suite:graphics, suite:graphics_per-day, suite:graphics_system"
+ATTRIBUTES = "suite:bvt-inline, suite:graphics, suite:graphics_per-day, suite:graphics_system"
PURPOSE = "Verify that graphics behaves as expected on idle."
CRITERIA = """
This test will fail if we don't see the appropriate GPU idle states.
diff --git a/client/site_tests/graphics_KernelMemory/control b/client/site_tests/graphics_KernelMemory/control
index 2fcbd9b..31d2729 100644
--- a/client/site_tests/graphics_KernelMemory/control
+++ b/client/site_tests/graphics_KernelMemory/control
@@ -15,9 +15,8 @@
TEST_CATEGORY = "Functional"
TEST_CLASS = "graphics"
TEST_TYPE = "client"
-JOB_RETRIES = 2
-ATTRIBUTES = "suite:bvt-cq, suite:graphics, suite:graphics_per-day, suite:graphics_system"
+ATTRIBUTES = "suite:bvt-perbuild, suite:graphics, suite:graphics_per-day, suite:graphics_system"
BUG_TEMPLATE = {
'components': ['OS>Kernel>Graphics'],
}
diff --git a/client/site_tests/graphics_LibDRM/control b/client/site_tests/graphics_LibDRM/control
index 01da2b0..b7d521d 100644
--- a/client/site_tests/graphics_LibDRM/control
+++ b/client/site_tests/graphics_LibDRM/control
@@ -12,9 +12,8 @@
TEST_CATEGORY = "Functional"
TEST_CLASS = "graphics"
TEST_TYPE = "client"
-JOB_RETRIES = 2
-ATTRIBUTES = "suite:bvt-cq, suite:graphics, suite:graphics_per-day, suite:graphics_system"
+ATTRIBUTES = "suite:bvt-perbuild, suite:graphics, suite:graphics_per-day, suite:graphics_system"
BUG_TEMPLATE = {
'components': ['OS>Kernel>Graphics'],
}
diff --git a/client/site_tests/graphics_PerfControl/control b/client/site_tests/graphics_PerfControl/control
index 82bc6cb..6bff9c7 100644
--- a/client/site_tests/graphics_PerfControl/control
+++ b/client/site_tests/graphics_PerfControl/control
@@ -15,7 +15,6 @@
TEST_CATEGORY = 'Performance'
TEST_CLASS = "Performance"
TEST_TYPE = 'client'
-JOB_RETRIES = 2
BUG_TEMPLATE = {
'components': ['OS>Kernel>Graphics'],
}
diff --git a/client/site_tests/graphics_SanAngeles/control.bvt-cq b/client/site_tests/graphics_SanAngeles/control.bvt
similarity index 92%
rename from client/site_tests/graphics_SanAngeles/control.bvt-cq
rename to client/site_tests/graphics_SanAngeles/control.bvt
index dabb420..da1b2c4 100644
--- a/client/site_tests/graphics_SanAngeles/control.bvt-cq
+++ b/client/site_tests/graphics_SanAngeles/control.bvt
@@ -3,10 +3,10 @@
# found in the LICENSE file.
AUTHOR = 'chromeos-gfx'
-NAME = 'graphics_SanAngeles.bvt-cq'
+NAME = 'graphics_SanAngeles.bvt'
PURPOSE = 'Benchmark OpenGL object rendering.'
CRITERIA = 'This test is a benchmark. It will fail if it fails to complete.'
-ATTRIBUTES = "suite:bvt-cq"
+ATTRIBUTES = "suite:bvt-perbuild"
TIME='FAST'
TEST_CATEGORY = 'Performance'
TEST_CLASS = "graphics"
diff --git a/client/site_tests/graphics_dEQP/control.bvt b/client/site_tests/graphics_dEQP/control.bvt
index e95c33c..e7497d9 100644
--- a/client/site_tests/graphics_dEQP/control.bvt
+++ b/client/site_tests/graphics_dEQP/control.bvt
@@ -8,7 +8,7 @@
AUTHOR = 'chromeos-gfx'
PURPOSE = 'Run the drawElements Quality Program test suite.'
CRITERIA = 'All of the individual tests must pass.'
-ATTRIBUTES = 'suite:deqp, suite:graphics_per-day, suite:graphics_system, suite:bvt-cq'
+ATTRIBUTES = 'suite:deqp, suite:graphics_per-day, suite:graphics_system, suite:bvt-inline'
TIME = 'FAST'
TEST_CATEGORY = 'Functional'
TEST_CLASS = 'graphics'
diff --git a/client/site_tests/graphics_dEQP/control.gles2.accuracy b/client/site_tests/graphics_dEQP/control.gles2.accuracy
deleted file mode 100644
index 237743e..0000000
--- a/client/site_tests/graphics_dEQP/control.gles2.accuracy
+++ /dev/null
@@ -1,26 +0,0 @@
-# Copyright 2015 The Chromium OS Authors. All rights reserved.
-# Use of this source code is governed by a BSD-style license that can be
-# found in the LICENSE file.
-
-# Please do not edit this file! It has been created by generate_controlfiles.py.
-
-NAME = 'graphics_dEQP.gles2.accuracy'
-AUTHOR = 'chromeos-gfx'
-PURPOSE = 'Run the drawElements Quality Program test suite.'
-CRITERIA = 'All of the individual tests must pass.'
-ATTRIBUTES = 'suite:deqp, suite:graphics_per-day, suite:graphics_system, suite:bvt-cq'
-TIME = 'FAST'
-TEST_CATEGORY = 'Functional'
-TEST_CLASS = 'graphics'
-TEST_TYPE = 'client'
-DOC = """
-This test runs the drawElements Quality Program test suite.
-"""
-job.run_test('graphics_dEQP',
- opts = args + [
- 'filter=dEQP-GLES2.accuracy',
- 'subset_to_run=Pass',
- 'hasty=False',
- 'shard_number=0',
- 'shard_count=1'
- ])
\ No newline at end of file
diff --git a/client/site_tests/graphics_dEQP/control.gles2.capability b/client/site_tests/graphics_dEQP/control.gles2.capability
deleted file mode 100644
index 77652a8..0000000
--- a/client/site_tests/graphics_dEQP/control.gles2.capability
+++ /dev/null
@@ -1,26 +0,0 @@
-# Copyright 2015 The Chromium OS Authors. All rights reserved.
-# Use of this source code is governed by a BSD-style license that can be
-# found in the LICENSE file.
-
-# Please do not edit this file! It has been created by generate_controlfiles.py.
-
-NAME = 'graphics_dEQP.gles2.capability'
-AUTHOR = 'chromeos-gfx'
-PURPOSE = 'Run the drawElements Quality Program test suite.'
-CRITERIA = 'All of the individual tests must pass.'
-ATTRIBUTES = 'suite:deqp, suite:graphics_per-day, suite:graphics_system, suite:bvt-cq'
-TIME = 'FAST'
-TEST_CATEGORY = 'Functional'
-TEST_CLASS = 'graphics'
-TEST_TYPE = 'client'
-DOC = """
-This test runs the drawElements Quality Program test suite.
-"""
-job.run_test('graphics_dEQP',
- opts = args + [
- 'filter=dEQP-GLES2.capability',
- 'subset_to_run=Pass',
- 'hasty=False',
- 'shard_number=0',
- 'shard_count=1'
- ])
\ No newline at end of file
diff --git a/client/site_tests/graphics_dEQP/control.gles2.info b/client/site_tests/graphics_dEQP/control.gles2.info
deleted file mode 100644
index f0a044d..0000000
--- a/client/site_tests/graphics_dEQP/control.gles2.info
+++ /dev/null
@@ -1,26 +0,0 @@
-# Copyright 2015 The Chromium OS Authors. All rights reserved.
-# Use of this source code is governed by a BSD-style license that can be
-# found in the LICENSE file.
-
-# Please do not edit this file! It has been created by generate_controlfiles.py.
-
-NAME = 'graphics_dEQP.gles2.info'
-AUTHOR = 'chromeos-gfx'
-PURPOSE = 'Run the drawElements Quality Program test suite.'
-CRITERIA = 'All of the individual tests must pass.'
-ATTRIBUTES = 'suite:deqp, suite:graphics_per-day, suite:graphics_system, suite:bvt-cq'
-TIME = 'FAST'
-TEST_CATEGORY = 'Functional'
-TEST_CLASS = 'graphics'
-TEST_TYPE = 'client'
-DOC = """
-This test runs the drawElements Quality Program test suite.
-"""
-job.run_test('graphics_dEQP',
- opts = args + [
- 'filter=dEQP-GLES2.info',
- 'subset_to_run=Pass',
- 'hasty=False',
- 'shard_number=0',
- 'shard_count=1'
- ])
\ No newline at end of file
diff --git a/client/site_tests/graphics_dEQP/control.gles2.performance b/client/site_tests/graphics_dEQP/control.gles2.performance
deleted file mode 100644
index 073a183..0000000
--- a/client/site_tests/graphics_dEQP/control.gles2.performance
+++ /dev/null
@@ -1,24 +0,0 @@
-# Copyright 2015 The Chromium OS Authors. All rights reserved.
-# Use of this source code is governed by a BSD-style license that can be
-# found in the LICENSE file.
-
-# Please do not edit this file! It has been created by generate_controlfiles.py.
-
-NAME = 'graphics_dEQP.gles2.performance'
-AUTHOR = 'chromeos-gfx'
-PURPOSE = 'Run the drawElements Quality Program test suite.'
-CRITERIA = 'All of the individual tests must pass.'
-ATTRIBUTES = 'suite:deqp, suite:graphics_per-day, suite:graphics_system'
-TIME = 'LONG'
-TEST_CATEGORY = 'Functional'
-TEST_CLASS = 'graphics'
-TEST_TYPE = 'client'
-DOC = """
-This test runs the drawElements Quality Program test suite.
-"""
-
-job.run_test('graphics_dEQP', opts = args + ['filter=dEQP-GLES2.performance',
- 'subset_to_run=Pass',
- 'hasty=False',
- 'shard_number=0',
- 'shard_count=1'])
\ No newline at end of file
diff --git a/client/site_tests/graphics_dEQP/control.gles2.stress b/client/site_tests/graphics_dEQP/control.gles2.stress
index f1016b8..37d084d 100644
--- a/client/site_tests/graphics_dEQP/control.gles2.stress
+++ b/client/site_tests/graphics_dEQP/control.gles2.stress
@@ -17,6 +17,7 @@
This test runs the drawElements Quality Program test suite.
"""
job.run_test('graphics_dEQP',
+ tag = 'stress',
opts = args + [
'filter=dEQP-GLES2.stress',
'subset_to_run=Pass',
diff --git a/client/site_tests/graphics_dEQP/control.gles3.accuracy b/client/site_tests/graphics_dEQP/control.gles3.accuracy
index 69a20e9..ffeb728 100644
--- a/client/site_tests/graphics_dEQP/control.gles3.accuracy
+++ b/client/site_tests/graphics_dEQP/control.gles3.accuracy
@@ -8,7 +8,7 @@
AUTHOR = 'chromeos-gfx'
PURPOSE = 'Run the drawElements Quality Program test suite.'
CRITERIA = 'All of the individual tests must pass.'
-ATTRIBUTES = 'suite:deqp, suite:graphics_per-day, suite:graphics_system, suite:bvt-cq'
+ATTRIBUTES = ''
TIME = 'FAST'
TEST_CATEGORY = 'Functional'
TEST_CLASS = 'graphics'
diff --git a/client/site_tests/graphics_dEQP/control.gles3.info b/client/site_tests/graphics_dEQP/control.gles3.info
deleted file mode 100644
index 5dfbdbf..0000000
--- a/client/site_tests/graphics_dEQP/control.gles3.info
+++ /dev/null
@@ -1,26 +0,0 @@
-# Copyright 2015 The Chromium OS Authors. All rights reserved.
-# Use of this source code is governed by a BSD-style license that can be
-# found in the LICENSE file.
-
-# Please do not edit this file! It has been created by generate_controlfiles.py.
-
-NAME = 'graphics_dEQP.gles3.info'
-AUTHOR = 'chromeos-gfx'
-PURPOSE = 'Run the drawElements Quality Program test suite.'
-CRITERIA = 'All of the individual tests must pass.'
-ATTRIBUTES = 'suite:deqp, suite:graphics_per-day, suite:graphics_system, suite:bvt-cq'
-TIME = 'FAST'
-TEST_CATEGORY = 'Functional'
-TEST_CLASS = 'graphics'
-TEST_TYPE = 'client'
-DOC = """
-This test runs the drawElements Quality Program test suite.
-"""
-job.run_test('graphics_dEQP',
- opts = args + [
- 'filter=dEQP-GLES3.info',
- 'subset_to_run=Pass',
- 'hasty=False',
- 'shard_number=0',
- 'shard_count=1'
- ])
\ No newline at end of file
diff --git a/client/site_tests/graphics_dEQP/control.gles3.performance b/client/site_tests/graphics_dEQP/control.gles3.performance
index 0c5ee43..f053526 100644
--- a/client/site_tests/graphics_dEQP/control.gles3.performance
+++ b/client/site_tests/graphics_dEQP/control.gles3.performance
@@ -8,7 +8,7 @@
AUTHOR = 'chromeos-gfx'
PURPOSE = 'Run the drawElements Quality Program test suite.'
CRITERIA = 'All of the individual tests must pass.'
-ATTRIBUTES = 'suite:deqp, suite:graphics_per-day, suite:graphics_system'
+ATTRIBUTES = ''
TIME = 'LONG'
TEST_CATEGORY = 'Functional'
TEST_CLASS = 'graphics'
diff --git a/client/site_tests/graphics_dEQP/control.gles3.stress b/client/site_tests/graphics_dEQP/control.gles3.stress
index 71df630..d2c0e19 100644
--- a/client/site_tests/graphics_dEQP/control.gles3.stress
+++ b/client/site_tests/graphics_dEQP/control.gles3.stress
@@ -8,7 +8,7 @@
AUTHOR = 'chromeos-gfx'
PURPOSE = 'Run the drawElements Quality Program test suite.'
CRITERIA = 'All of the individual tests must pass.'
-ATTRIBUTES = 'suite:deqp, suite:graphics_per-day, suite:graphics_system'
+ATTRIBUTES = ''
TIME = 'LONG'
TEST_CATEGORY = 'Functional'
TEST_CLASS = 'graphics'
diff --git a/client/site_tests/graphics_dEQP/control.gles31.info b/client/site_tests/graphics_dEQP/control.gles31.info
deleted file mode 100644
index 4e47244..0000000
--- a/client/site_tests/graphics_dEQP/control.gles31.info
+++ /dev/null
@@ -1,26 +0,0 @@
-# Copyright 2015 The Chromium OS Authors. All rights reserved.
-# Use of this source code is governed by a BSD-style license that can be
-# found in the LICENSE file.
-
-# Please do not edit this file! It has been created by generate_controlfiles.py.
-
-NAME = 'graphics_dEQP.gles31.info'
-AUTHOR = 'chromeos-gfx'
-PURPOSE = 'Run the drawElements Quality Program test suite.'
-CRITERIA = 'All of the individual tests must pass.'
-ATTRIBUTES = 'suite:deqp, suite:graphics_per-day, suite:graphics_system, suite:bvt-cq'
-TIME = 'FAST'
-TEST_CATEGORY = 'Functional'
-TEST_CLASS = 'graphics'
-TEST_TYPE = 'client'
-DOC = """
-This test runs the drawElements Quality Program test suite.
-"""
-job.run_test('graphics_dEQP',
- opts = args + [
- 'filter=dEQP-GLES31.info',
- 'subset_to_run=Pass',
- 'hasty=False',
- 'shard_number=0',
- 'shard_count=1'
- ])
\ No newline at end of file
diff --git a/client/site_tests/graphics_dEQP/control.gles31.stress b/client/site_tests/graphics_dEQP/control.gles31.stress
index 6becde3..a8f794e 100644
--- a/client/site_tests/graphics_dEQP/control.gles31.stress
+++ b/client/site_tests/graphics_dEQP/control.gles31.stress
@@ -8,7 +8,7 @@
AUTHOR = 'chromeos-gfx'
PURPOSE = 'Run the drawElements Quality Program test suite.'
CRITERIA = 'All of the individual tests must pass.'
-ATTRIBUTES = ''
+ATTRIBUTES = 'suite:deqp, suite:graphics_per-day, suite:graphics_system'
TIME = 'LONG'
TEST_CATEGORY = 'Functional'
TEST_CLASS = 'graphics'
@@ -17,6 +17,7 @@
This test runs the drawElements Quality Program test suite.
"""
job.run_test('graphics_dEQP',
+ tag = 'stress',
opts = args + [
'filter=dEQP-GLES31.stress',
'subset_to_run=Pass',
diff --git a/client/site_tests/graphics_dEQP/control.vk-master.hasty b/client/site_tests/graphics_dEQP/control.vk-master.hasty
index a194779..5430f6d 100644
--- a/client/site_tests/graphics_dEQP/control.vk-master.hasty
+++ b/client/site_tests/graphics_dEQP/control.vk-master.hasty
@@ -8,7 +8,7 @@
CRITERIA = 'All of the individual tests must pass.'
# We run vk-master.hasty by default as vk-master takes 12h and VK is expected
# to pass in a single process run.
-ATTRIBUTES = 'suite:deqp, suite:graphics_per-day, suite:graphics_system'
+ATTRIBUTES = 'suite:bvt-perbuild, suite:deqp, suite:graphics_per-day, suite:graphics_system'
TIME = 'LENGTHY'
TEST_CATEGORY = 'Functional'
TEST_CLASS = 'graphics'
diff --git a/client/site_tests/graphics_dEQP/control.vk-master.hasty.0 b/client/site_tests/graphics_dEQP/control.vk-master.hasty.0
index fc29960..1c009e1 100644
--- a/client/site_tests/graphics_dEQP/control.vk-master.hasty.0
+++ b/client/site_tests/graphics_dEQP/control.vk-master.hasty.0
@@ -8,7 +8,7 @@
AUTHOR = 'chromeos-gfx'
PURPOSE = 'Run the drawElements Quality Program test suite.'
CRITERIA = 'All of the individual tests must pass.'
-ATTRIBUTES = 'suite:deqp, suite:graphics_per-day, suite:graphics_system'
+ATTRIBUTES = ''
TIME = 'FAST'
TEST_CATEGORY = 'Functional'
TEST_CLASS = 'graphics'
diff --git a/client/site_tests/graphics_dEQP/control.vk-master.hasty.1 b/client/site_tests/graphics_dEQP/control.vk-master.hasty.1
index 8b48bba..33e54f8 100644
--- a/client/site_tests/graphics_dEQP/control.vk-master.hasty.1
+++ b/client/site_tests/graphics_dEQP/control.vk-master.hasty.1
@@ -8,7 +8,7 @@
AUTHOR = 'chromeos-gfx'
PURPOSE = 'Run the drawElements Quality Program test suite.'
CRITERIA = 'All of the individual tests must pass.'
-ATTRIBUTES = 'suite:deqp, suite:graphics_per-day, suite:graphics_system'
+ATTRIBUTES = ''
TIME = 'FAST'
TEST_CATEGORY = 'Functional'
TEST_CLASS = 'graphics'
diff --git a/client/site_tests/graphics_dEQP/control.vk-master.hasty.2 b/client/site_tests/graphics_dEQP/control.vk-master.hasty.2
index e1dad32..f9310b9 100644
--- a/client/site_tests/graphics_dEQP/control.vk-master.hasty.2
+++ b/client/site_tests/graphics_dEQP/control.vk-master.hasty.2
@@ -8,7 +8,7 @@
AUTHOR = 'chromeos-gfx'
PURPOSE = 'Run the drawElements Quality Program test suite.'
CRITERIA = 'All of the individual tests must pass.'
-ATTRIBUTES = 'suite:deqp, suite:graphics_per-day, suite:graphics_system'
+ATTRIBUTES = ''
TIME = 'FAST'
TEST_CATEGORY = 'Functional'
TEST_CLASS = 'graphics'
diff --git a/client/site_tests/graphics_dEQP/control.vk-master.hasty.3 b/client/site_tests/graphics_dEQP/control.vk-master.hasty.3
index 1ff263e..aaec78e 100644
--- a/client/site_tests/graphics_dEQP/control.vk-master.hasty.3
+++ b/client/site_tests/graphics_dEQP/control.vk-master.hasty.3
@@ -8,7 +8,7 @@
AUTHOR = 'chromeos-gfx'
PURPOSE = 'Run the drawElements Quality Program test suite.'
CRITERIA = 'All of the individual tests must pass.'
-ATTRIBUTES = 'suite:deqp, suite:graphics_per-day, suite:graphics_system'
+ATTRIBUTES = ''
TIME = 'FAST'
TEST_CATEGORY = 'Functional'
TEST_CLASS = 'graphics'
diff --git a/client/site_tests/graphics_dEQP/control.vk-master.hasty.4 b/client/site_tests/graphics_dEQP/control.vk-master.hasty.4
index 3b63ef4..fedbfa3 100644
--- a/client/site_tests/graphics_dEQP/control.vk-master.hasty.4
+++ b/client/site_tests/graphics_dEQP/control.vk-master.hasty.4
@@ -8,7 +8,7 @@
AUTHOR = 'chromeos-gfx'
PURPOSE = 'Run the drawElements Quality Program test suite.'
CRITERIA = 'All of the individual tests must pass.'
-ATTRIBUTES = 'suite:deqp, suite:graphics_per-day, suite:graphics_system'
+ATTRIBUTES = ''
TIME = 'FAST'
TEST_CATEGORY = 'Functional'
TEST_CLASS = 'graphics'
diff --git a/client/site_tests/graphics_dEQP/control.vk-master.hasty.5 b/client/site_tests/graphics_dEQP/control.vk-master.hasty.5
index 25d5825..78ed5f0 100644
--- a/client/site_tests/graphics_dEQP/control.vk-master.hasty.5
+++ b/client/site_tests/graphics_dEQP/control.vk-master.hasty.5
@@ -8,7 +8,7 @@
AUTHOR = 'chromeos-gfx'
PURPOSE = 'Run the drawElements Quality Program test suite.'
CRITERIA = 'All of the individual tests must pass.'
-ATTRIBUTES = 'suite:deqp, suite:graphics_per-day, suite:graphics_system'
+ATTRIBUTES = ''
TIME = 'FAST'
TEST_CATEGORY = 'Functional'
TEST_CLASS = 'graphics'
diff --git a/client/site_tests/graphics_dEQP/control.vk-master.hasty.6 b/client/site_tests/graphics_dEQP/control.vk-master.hasty.6
index 7755e5b..69b6107 100644
--- a/client/site_tests/graphics_dEQP/control.vk-master.hasty.6
+++ b/client/site_tests/graphics_dEQP/control.vk-master.hasty.6
@@ -8,7 +8,7 @@
AUTHOR = 'chromeos-gfx'
PURPOSE = 'Run the drawElements Quality Program test suite.'
CRITERIA = 'All of the individual tests must pass.'
-ATTRIBUTES = 'suite:deqp, suite:graphics_per-day, suite:graphics_system'
+ATTRIBUTES = ''
TIME = 'FAST'
TEST_CATEGORY = 'Functional'
TEST_CLASS = 'graphics'
diff --git a/client/site_tests/graphics_dEQP/control.vk-master.hasty.7 b/client/site_tests/graphics_dEQP/control.vk-master.hasty.7
index dc6a4e8..f87b29b 100644
--- a/client/site_tests/graphics_dEQP/control.vk-master.hasty.7
+++ b/client/site_tests/graphics_dEQP/control.vk-master.hasty.7
@@ -8,7 +8,7 @@
AUTHOR = 'chromeos-gfx'
PURPOSE = 'Run the drawElements Quality Program test suite.'
CRITERIA = 'All of the individual tests must pass.'
-ATTRIBUTES = 'suite:deqp, suite:graphics_per-day, suite:graphics_system'
+ATTRIBUTES = ''
TIME = 'FAST'
TEST_CATEGORY = 'Functional'
TEST_CLASS = 'graphics'
diff --git a/client/site_tests/graphics_dEQP/control.vk-master.hasty.8 b/client/site_tests/graphics_dEQP/control.vk-master.hasty.8
index f1cf574..3bcfe2f 100644
--- a/client/site_tests/graphics_dEQP/control.vk-master.hasty.8
+++ b/client/site_tests/graphics_dEQP/control.vk-master.hasty.8
@@ -8,7 +8,7 @@
AUTHOR = 'chromeos-gfx'
PURPOSE = 'Run the drawElements Quality Program test suite.'
CRITERIA = 'All of the individual tests must pass.'
-ATTRIBUTES = 'suite:deqp, suite:graphics_per-day, suite:graphics_system'
+ATTRIBUTES = ''
TIME = 'FAST'
TEST_CATEGORY = 'Functional'
TEST_CLASS = 'graphics'
diff --git a/client/site_tests/graphics_dEQP/control.vk-master.hasty.9 b/client/site_tests/graphics_dEQP/control.vk-master.hasty.9
index 660ec9b..d4d8e97 100644
--- a/client/site_tests/graphics_dEQP/control.vk-master.hasty.9
+++ b/client/site_tests/graphics_dEQP/control.vk-master.hasty.9
@@ -8,7 +8,7 @@
AUTHOR = 'chromeos-gfx'
PURPOSE = 'Run the drawElements Quality Program test suite.'
CRITERIA = 'All of the individual tests must pass.'
-ATTRIBUTES = 'suite:deqp, suite:graphics_per-day, suite:graphics_system'
+ATTRIBUTES = ''
TIME = 'FAST'
TEST_CATEGORY = 'Functional'
TEST_CLASS = 'graphics'
diff --git a/client/site_tests/graphics_dEQP/control.vk.api.hasty b/client/site_tests/graphics_dEQP/control.vk.api.hasty
deleted file mode 100644
index 109eb19..0000000
--- a/client/site_tests/graphics_dEQP/control.vk.api.hasty
+++ /dev/null
@@ -1,26 +0,0 @@
-# Copyright 2015 The Chromium OS Authors. All rights reserved.
-# Use of this source code is governed by a BSD-style license that can be
-# found in the LICENSE file.
-
-# Please do not edit this file! It has been created by generate_controlfiles.py.
-
-NAME = 'graphics_dEQP.vk.api.hasty'
-AUTHOR = 'chromeos-gfx'
-PURPOSE = 'Run the drawElements Quality Program test suite.'
-CRITERIA = 'All of the individual tests must pass.'
-ATTRIBUTES = ''
-TIME = 'LONG'
-TEST_CATEGORY = 'Functional'
-TEST_CLASS = 'graphics'
-TEST_TYPE = 'client'
-DOC = """
-This test runs the drawElements Quality Program test suite.
-"""
-job.run_test('graphics_dEQP',
- opts = args + [
- 'filter=dEQP-VK.api',
- 'subset_to_run=Pass',
- 'hasty=True',
- 'shard_number=0',
- 'shard_count=1'
- ])
\ No newline at end of file
diff --git a/client/site_tests/graphics_dEQP/control.vk.api.smoke b/client/site_tests/graphics_dEQP/control.vk.api.smoke
deleted file mode 100644
index 1b2e6b6..0000000
--- a/client/site_tests/graphics_dEQP/control.vk.api.smoke
+++ /dev/null
@@ -1,26 +0,0 @@
-# Copyright 2015 The Chromium OS Authors. All rights reserved.
-# Use of this source code is governed by a BSD-style license that can be
-# found in the LICENSE file.
-
-# Please do not edit this file! It has been created by generate_controlfiles.py.
-
-NAME = 'graphics_dEQP.vk.api.smoke'
-AUTHOR = 'chromeos-gfx'
-PURPOSE = 'Run the drawElements Quality Program test suite.'
-CRITERIA = 'All of the individual tests must pass.'
-ATTRIBUTES = 'suite:deqp, suite:graphics_per-day, suite:graphics_system, suite:bvt-cq'
-TIME = 'FAST'
-TEST_CATEGORY = 'Functional'
-TEST_CLASS = 'graphics'
-TEST_TYPE = 'client'
-DOC = """
-This test runs the drawElements Quality Program test suite.
-"""
-job.run_test('graphics_dEQP',
- opts = args + [
- 'filter=dEQP-VK.api.smoke',
- 'subset_to_run=Pass',
- 'hasty=False',
- 'shard_number=0',
- 'shard_count=1'
- ])
\ No newline at end of file
diff --git a/client/site_tests/graphics_dEQP/control.vk.binding_model.hasty b/client/site_tests/graphics_dEQP/control.vk.binding_model.hasty
deleted file mode 100644
index 8d5d5b9..0000000
--- a/client/site_tests/graphics_dEQP/control.vk.binding_model.hasty
+++ /dev/null
@@ -1,26 +0,0 @@
-# Copyright 2015 The Chromium OS Authors. All rights reserved.
-# Use of this source code is governed by a BSD-style license that can be
-# found in the LICENSE file.
-
-# Please do not edit this file! It has been created by generate_controlfiles.py.
-
-NAME = 'graphics_dEQP.vk.binding_model.hasty'
-AUTHOR = 'chromeos-gfx'
-PURPOSE = 'Run the drawElements Quality Program test suite.'
-CRITERIA = 'All of the individual tests must pass.'
-ATTRIBUTES = ''
-TIME = 'LONG'
-TEST_CATEGORY = 'Functional'
-TEST_CLASS = 'graphics'
-TEST_TYPE = 'client'
-DOC = """
-This test runs the drawElements Quality Program test suite.
-"""
-job.run_test('graphics_dEQP',
- opts = args + [
- 'filter=dEQP-VK.binding_model',
- 'subset_to_run=Pass',
- 'hasty=True',
- 'shard_number=0',
- 'shard_count=1'
- ])
\ No newline at end of file
diff --git a/client/site_tests/graphics_dEQP/control.vk.glsl.hasty b/client/site_tests/graphics_dEQP/control.vk.glsl.hasty
deleted file mode 100644
index 4ca9a32..0000000
--- a/client/site_tests/graphics_dEQP/control.vk.glsl.hasty
+++ /dev/null
@@ -1,26 +0,0 @@
-# Copyright 2015 The Chromium OS Authors. All rights reserved.
-# Use of this source code is governed by a BSD-style license that can be
-# found in the LICENSE file.
-
-# Please do not edit this file! It has been created by generate_controlfiles.py.
-
-NAME = 'graphics_dEQP.vk.glsl.hasty'
-AUTHOR = 'chromeos-gfx'
-PURPOSE = 'Run the drawElements Quality Program test suite.'
-CRITERIA = 'All of the individual tests must pass.'
-ATTRIBUTES = ''
-TIME = 'LONG'
-TEST_CATEGORY = 'Functional'
-TEST_CLASS = 'graphics'
-TEST_TYPE = 'client'
-DOC = """
-This test runs the drawElements Quality Program test suite.
-"""
-job.run_test('graphics_dEQP',
- opts = args + [
- 'filter=dEQP-VK.glsl',
- 'subset_to_run=Pass',
- 'hasty=True',
- 'shard_number=0',
- 'shard_count=1'
- ])
\ No newline at end of file
diff --git a/client/site_tests/graphics_dEQP/control.vk.info b/client/site_tests/graphics_dEQP/control.vk.info
deleted file mode 100644
index 638fed8..0000000
--- a/client/site_tests/graphics_dEQP/control.vk.info
+++ /dev/null
@@ -1,26 +0,0 @@
-# Copyright 2015 The Chromium OS Authors. All rights reserved.
-# Use of this source code is governed by a BSD-style license that can be
-# found in the LICENSE file.
-
-# Please do not edit this file! It has been created by generate_controlfiles.py.
-
-NAME = 'graphics_dEQP.vk.info'
-AUTHOR = 'chromeos-gfx'
-PURPOSE = 'Run the drawElements Quality Program test suite.'
-CRITERIA = 'All of the individual tests must pass.'
-ATTRIBUTES = 'suite:deqp, suite:graphics_per-day, suite:graphics_system, suite:bvt-cq'
-TIME = 'FAST'
-TEST_CATEGORY = 'Functional'
-TEST_CLASS = 'graphics'
-TEST_TYPE = 'client'
-DOC = """
-This test runs the drawElements Quality Program test suite.
-"""
-job.run_test('graphics_dEQP',
- opts = args + [
- 'filter=dEQP-VK.info',
- 'subset_to_run=Pass',
- 'hasty=False',
- 'shard_number=0',
- 'shard_count=1'
- ])
\ No newline at end of file
diff --git a/client/site_tests/graphics_dEQP/control.vk.pipeline.hasty b/client/site_tests/graphics_dEQP/control.vk.pipeline.hasty
deleted file mode 100644
index b70936c..0000000
--- a/client/site_tests/graphics_dEQP/control.vk.pipeline.hasty
+++ /dev/null
@@ -1,26 +0,0 @@
-# Copyright 2015 The Chromium OS Authors. All rights reserved.
-# Use of this source code is governed by a BSD-style license that can be
-# found in the LICENSE file.
-
-# Please do not edit this file! It has been created by generate_controlfiles.py.
-
-NAME = 'graphics_dEQP.vk.pipeline.hasty'
-AUTHOR = 'chromeos-gfx'
-PURPOSE = 'Run the drawElements Quality Program test suite.'
-CRITERIA = 'All of the individual tests must pass.'
-ATTRIBUTES = ''
-TIME = 'LONG'
-TEST_CATEGORY = 'Functional'
-TEST_CLASS = 'graphics'
-TEST_TYPE = 'client'
-DOC = """
-This test runs the drawElements Quality Program test suite.
-"""
-job.run_test('graphics_dEQP',
- opts = args + [
- 'filter=dEQP-VK.pipeline',
- 'subset_to_run=Pass',
- 'hasty=True',
- 'shard_number=0',
- 'shard_count=1'
- ])
\ No newline at end of file
diff --git a/client/site_tests/graphics_dEQP/control.vk.spirv_assembly.hasty b/client/site_tests/graphics_dEQP/control.vk.spirv_assembly.hasty
deleted file mode 100644
index 26968ea..0000000
--- a/client/site_tests/graphics_dEQP/control.vk.spirv_assembly.hasty
+++ /dev/null
@@ -1,26 +0,0 @@
-# Copyright 2015 The Chromium OS Authors. All rights reserved.
-# Use of this source code is governed by a BSD-style license that can be
-# found in the LICENSE file.
-
-# Please do not edit this file! It has been created by generate_controlfiles.py.
-
-NAME = 'graphics_dEQP.vk.spirv_assembly.hasty'
-AUTHOR = 'chromeos-gfx'
-PURPOSE = 'Run the drawElements Quality Program test suite.'
-CRITERIA = 'All of the individual tests must pass.'
-ATTRIBUTES = ''
-TIME = 'SHORT'
-TEST_CATEGORY = 'Functional'
-TEST_CLASS = 'graphics'
-TEST_TYPE = 'client'
-DOC = """
-This test runs the drawElements Quality Program test suite.
-"""
-job.run_test('graphics_dEQP',
- opts = args + [
- 'filter=dEQP-VK.spirv_assembly',
- 'subset_to_run=Pass',
- 'hasty=True',
- 'shard_number=0',
- 'shard_count=1'
- ])
\ No newline at end of file
diff --git a/client/site_tests/graphics_dEQP/generate_controlfiles.py b/client/site_tests/graphics_dEQP/generate_controlfiles.py
index 14ac968..2d3771c 100755
--- a/client/site_tests/graphics_dEQP/generate_controlfiles.py
+++ b/client/site_tests/graphics_dEQP/generate_controlfiles.py
@@ -19,7 +19,7 @@
ATTRIBUTES_BVT_CQ = (
- 'suite:deqp, suite:graphics_per-day, suite:graphics_system, suite:bvt-cq')
+ 'suite:deqp, suite:graphics_per-day, suite:graphics_system, suite:bvt-inline')
ATTRIBUTES_BVT_PB = (
'suite:deqp, suite:graphics_per-day, suite:graphics_system, '
'suite:bvt-perbuild'
@@ -43,29 +43,21 @@
Test('bvt', Suite.bvtcq, shards=1, hasty=False, time='FAST', tag='bvt', test_file=BVT_MASTER_FILE),
Test('dEQP-GLES2-master', Suite.daily, shards=1, hasty=False, time='LENGTHY', tag='gles2-master', test_file=GLES2_MASTER_FILE),
Test('dEQP-GLES2-master', Suite.bvtpb, shards=10, hasty=True, time='FAST', tag='gles2-master', test_file=GLES2_MASTER_FILE),
- Test('dEQP-GLES2.accuracy', Suite.bvtcq, shards=1, hasty=False, time='FAST', tag=None, test_file=None),
- Test('dEQP-GLES2.capability', Suite.bvtcq, shards=1, hasty=False, time='FAST', tag=None, test_file=None),
- Test('dEQP-GLES2.info', Suite.bvtcq, shards=1, hasty=False, time='FAST', tag=None, test_file=None),
- Test('dEQP-GLES2.stress', Suite.daily, shards=1, hasty=False, time='LONG', tag=None, test_file=None),
- Test('dEQP-GLES3.accuracy', Suite.bvtcq, shards=1, hasty=False, time='FAST', tag=None, test_file=None),
+ # The stress, accuracy and performance tests are not part of -master lists.
+ # Hence we create control files in case we want to run them. But there is
+ # no strict requirement to keep them passing.
+ Test('dEQP-GLES2.stress', Suite.daily, shards=1, hasty=False, time='LONG', tag='stress', test_file=None),
+ Test('dEQP-GLES3.accuracy', Suite.none, shards=1, hasty=False, time='FAST', tag=None, test_file=None),
Test('dEQP-GLES3-master', Suite.daily, shards=1, hasty=False, time='LENGTHY', tag='gles3-master', test_file=GLES3_MASTER_FILE),
Test('dEQP-GLES3-master', Suite.bvtpb, shards=10, hasty=True, time='FAST', tag='gles3-master', test_file=GLES3_MASTER_FILE),
- Test('dEQP-GLES3.info', Suite.bvtcq, shards=1, hasty=False, time='FAST', tag=None, test_file=None),
- Test('dEQP-GLES3.performance', Suite.daily, shards=1, hasty=False, time='LONG', tag=None, test_file=None),
- Test('dEQP-GLES3.stress', Suite.daily, shards=1, hasty=False, time='LONG', tag=None, test_file=None),
+ Test('dEQP-GLES3.performance', Suite.none, shards=1, hasty=False, time='LONG', tag=None, test_file=None),
+ Test('dEQP-GLES3.stress', Suite.none, shards=1, hasty=False, time='LONG', tag=None, test_file=None),
Test('dEQP-GLES31-master', Suite.daily, shards=1, hasty=False, time='LENGTHY', tag='gles31-master', test_file=GLES31_MASTER_FILE),
Test('dEQP-GLES31-master', Suite.bvtpb, shards=10, hasty=True, time='FAST', tag='gles31-master', test_file=GLES31_MASTER_FILE),
- Test('dEQP-GLES31.info', Suite.bvtcq, shards=1, hasty=False, time='FAST', tag=None, test_file=None),
- Test('dEQP-GLES31.stress', Suite.none, shards=1, hasty=False, time='LONG', tag=None, test_file=None),
+ Test('dEQP-GLES31.stress', Suite.daily, shards=1, hasty=False, time='LONG', tag='stress', test_file=None),
Test('dEQP-VK-master', Suite.none, shards=1, hasty=False, time='LENGTHY', tag='vk-master', test_file=VK_MASTER_FILE),
- Test('dEQP-VK-master', Suite.daily, shards=10, hasty=True, time='FAST', tag='vk-master', test_file=VK_MASTER_FILE),
- Test('dEQP-VK.api', Suite.none, shards=1, hasty=True, time='LONG', tag=None, test_file=None),
- Test('dEQP-VK.api.smoke', Suite.bvtcq, shards=1, hasty=False, time='FAST', tag=None, test_file=None),
- Test('dEQP-VK.binding_model', Suite.none, shards=1, hasty=True, time='LONG', tag=None, test_file=None),
- Test('dEQP-VK.glsl', Suite.none, shards=1, hasty=True, time='LONG', tag=None, test_file=None),
- Test('dEQP-VK.info', Suite.bvtcq, shards=1, hasty=False, time='FAST', tag=None, test_file=None),
- Test('dEQP-VK.pipeline', Suite.none, shards=1, hasty=True, time='LONG', tag=None, test_file=None),
- Test('dEQP-VK.spirv_assembly', Suite.none, shards=1, hasty=True, time='SHORT', tag=None, test_file=None),
+ # TODO(pwang): fix VK-master.hasty
+ Test('dEQP-VK-master', Suite.none, shards=10, hasty=True, time='FAST', tag='vk-master', test_file=VK_MASTER_FILE),
]
CONTROLFILE_TEMPLATE = Template(
