moblab_RunSuite: Retry tests launched on moblab
moblab's sub-DUTs are often unreliable, or at least, are as likely to
fail provision as any other DUT. We retry tests in suites launched from
the builders in order to avoid the suite failing due to a small number
of provision failures. By the same logic, we should retry the tests
inside the suite launched on moblab: to avoid failing the test when a
small number of provisions fail.
At the same time, keep the retry limit small: We don't have that many
DUTs attached to each moblab instance, so retrying tests directly
translates to longer end-to-end moblab_RunSuite time.
BUG=chromium:729099
TEST=moblab_RunSuite
Change-Id: If528c95241ca422f7f0ed8d17984ac37eff0f430
Reviewed-on: https://chromium-review.googlesource.com/522926
Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org>
Tested-by: Prathmesh Prabhu <pprabhu@chromium.org>
Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org>
Reviewed-by: Simran Basi <sbasi@chromium.org>
diff --git a/server/site_tests/moblab_RunSuite/control.dummyServer b/server/site_tests/moblab_RunSuite/control.dummyServer
index 313ed9f..9b7c4e1 100644
--- a/server/site_tests/moblab_RunSuite/control.dummyServer
+++ b/server/site_tests/moblab_RunSuite/control.dummyServer
@@ -37,6 +37,6 @@
host = hosts.create_host(machine)
args_dict = utils.args_to_dict(args)
job.run_test('moblab_RunSuite', host=host, suite_name='dummy_server',
- **args_dict)
+ moblab_suite_max_retries=1, **args_dict)
parallel_simple(run, machines)
diff --git a/server/site_tests/moblab_RunSuite/control.smoke b/server/site_tests/moblab_RunSuite/control.smoke
index d1fca39..1b75bc4 100644
--- a/server/site_tests/moblab_RunSuite/control.smoke
+++ b/server/site_tests/moblab_RunSuite/control.smoke
@@ -35,6 +35,7 @@
def run(machine):
host = hosts.create_host(machine)
args_dict = utils.args_to_dict(args)
- job.run_test('moblab_RunSuite', host=host, suite_name='smoke', **args_dict)
+ job.run_test('moblab_RunSuite', host=host, suite_name='smoke',
+ moblab_suite_max_retries=1, **args_dict)
parallel_simple(run, machines)
diff --git a/server/site_tests/moblab_RunSuite/moblab_RunSuite.py b/server/site_tests/moblab_RunSuite/moblab_RunSuite.py
index a8262a4..de783ff 100644
--- a/server/site_tests/moblab_RunSuite/moblab_RunSuite.py
+++ b/server/site_tests/moblab_RunSuite/moblab_RunSuite.py
@@ -22,11 +22,13 @@
version = 1
- def run_once(self, host, suite_name):
+ def run_once(self, host, suite_name, moblab_suite_max_retries):
"""Runs a suite on a Moblab Host against its test DUTS.
@param host: Moblab Host that will run the suite.
@param suite_name: Name of the suite to run.
+ @param moblab_suite_max_retries: The maximum number of test retries
+ allowed within the suite launched on moblab.
@raises AutoservRunError if the suite does not complete successfully.
"""
@@ -47,9 +49,10 @@
try:
result = host.run_as_moblab(
"%s/site_utils/run_suite.py --pool='' "
- "--board=%s --build=%s --suite_name=%s" %
+ "--board=%s --build=%s --suite_name=%s --retry=True "
+ "--max_retries=%d" %
(moblab_host.AUTOTEST_INSTALL_DIR, board, build,
- suite_name), timeout=10800)
+ suite_name, moblab_suite_max_retries), timeout=10800)
except error.AutoservRunError as e:
# Collect the results and logs from the moblab device.
moblab_logs_dir = os.path.join(self.resultsdir, 'moblab_logs')
