en/compatibility/vts/multi-device-testing.html - platform/docs/source.android.com - Git at Google

 <html devsite>
   <head>
     <title>Multi-Device Testing</title>
     <meta name="project_path" value="/_project.yaml" />
     <meta name="book_path" value="/_book.yaml" />
   </head>
   <body>
   <!--
       Copyright 2018 The Android Open Source Project

       Licensed under the Apache License, Version 2.0 (the "License");
       you may not use this file except in compliance with the License.
       You may obtain a copy of the License at

           http://www.apache.org/licenses/LICENSE-2.0

       Unless required by applicable law or agreed to in writing, software
       distributed under the License is distributed on an "AS IS" BASIS,
       WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
       See the License for the specific language governing permissions and
       limitations under the License.
   -->


 <p>VTS supports tests that require interaction between multiple Android
 devices.</p>

 <h2 id="architecture">Architecture</h2>
 <p>VTS uses the TradeFed framework to get and pass device serials to test
 modules.</p>

 <p><img src="images/vts_device_serials.png"></p>
 <figcaption><strong>Figure 1.</strong> VTS passing device serials.</figcaption>

 <p>Device requirements, such as number of devices and device types, are
 specified in test plan configuration. For example, you can specify a test plan
 that requires two Android devices with Sailfish build targets.</p>

 <h3 id="device-allocation">Device allocation</h3>
 <p>The test infrastructure (usually the test scheduler) allocates available
 devices that satisfy the requirements specified in test plan configuration to
 the VTS framework. Allocated devices are reserved for the test plan even if the
 test module is not using them. VTS agent binaries are then pushed to and run on
 all allocated devices (unless specifically instructed not to run). This ensures
 that TCP connections for shell commands and HAL RPCs are available for all
 devices in a test script.</p>

 <h3 id="test-preparers">Test preparers</h3>
 <p>The framework runs test preparers for all devices for which it received
 serial numbers. Target preparers can be single or multi-device:</p>
 <ul>
 <li>Single-device target preparers (example at
 <a href="https://android.googlesource.com/platform/test/vts/+/master/harnesses/tradefed/src/com/android/tradefed/targetprep/VtsDeviceInfoCollector.java" class="external">VtsDeviceInfoCollector</a>):
 <ul>
 <li>Can be specified only in test plan configuration with the required
 device list (future versions will allow module level configuration).</li>
 <li>Receive only one device serial.</li>
 <li>Run preparing and cleanup tasks against a specific device.</li>
 </ul>
 </li>
 <li>Multi-device target preparers (example at
 <a href="https://android.googlesource.com/platform/test/vts/+/master/harnesses/tradefed/src/com/android/tradefed/targetprep/VtsPythonVirtualenvPreparer.java" class="external">VtsPythonVirtualenvPreparer</a>):
 <ul>
 <li>Can be specified in test plan configuration or test module
 configuration</li>
 <li>Receive all device serials</li>
 <li>Run preparing and cleanup tasks for each device or all devices.</li>
 </ul>
 </li>
 </ul>

 <h3 id="test-modules">Test modules</h3>
 <p>Test modules get a list of devices after the test preparers finish setting up
 the host/devices. One host-side Python test module runs for each multi-device
 test module. Allocated Android devices are accessible from Python test modules
 as a list of
 <a href="https://android.googlesource.com/platform/test/vts/+/master/utils/python/controllers/android_device.py#322" class="external">AndroidDevice</a>
 objects:</p>
 <pre class="devsite-click-to-copy">
 devices = self.android_devices
 device1 = devices[0]
 device1_serial = device1.serial
 </pre>

 <p>All allocated devices are reserved for the test plan, even though a test
 module in the plan is only using one device.</p>

 <h2 id="device-communication">Device communication during testing</h1>
 <p>Effective multi-Android tests involve communication between allocated
 devices. When developing such tests, you must determine how to establish
 communication between the allocated devices. The following sections provide
 three communication examples (however, test developers are free to design other
 models).</p>

 <h3 id="type1">Type 1: Host-side HAL tests</h3>
 <p>Host-side HAL tests can use VTS HAL drivers that are pushed to devices by
 default:</p>

 <p><img src="images/vts_hostside_hal.png"></p>
 <figcaption><strong>Figure 2.</strong> Host-side HAL test.</figcaption>

 <p>In this scenario:</p>
 <ul>
 <li>Test logic executes on the host.</li>
 <li>Host-side test script issues RPC calls to the drivers on each device.</li>
 <li>Host side coordinates device interactions.</li>
 </ul>

 <h3 id="type2">Type 2: Host-side agent-based tests</h3>
 <p>Instead of using VTS agents on device, a host-side test can also push its own
 agent (app or binary) to each device:</p>

 <p><img src="images/vts_hostside_agent.png"></p>
 <figcaption><strong>Figure 3.</strong> Host-side, agent-based test.</figcaption>

 <p>In this scenario:</p>
 <ul>
 <li>Test logic executes on the host.</li>
 <li>Agent app (or binary) installs on each device.</li>
 <li>Host-side test script issues commands to apps on each device.</li>
 <li>Host side coordinates device interactions.</li>
 </ul>

 <p>For example, the
 <a href="https://android.googlesource.com/platform/test/vts-testcase/nbu/+/master" class="external">Next
 Billion User tests</a> in current VTS repo are host-side, app-based,
 multi-device tests.</p>

 <h3 id="type3">Type 3: Target-side HIDL tests</h2>
 <p>Target-side, multi-device HIDL tests put all test logic on device-side test
 binaries, which requires the tests to synchronize devices during test
 execution:</p>

 <p><img src="images/vts_target_hidl.png"></p>
 <figcaption><strong>Figure 4. </strong>Target-based HIDL test.</figcaption>

 <p>In this scenario:</p>
 <ul>
 <li>Test logic executes on devices.</li>
 <li>Host-side framework provides initial device identification.</li>
 <li>Target-side test binary requires synchronization:<ul>
 <li>Same test binary for all devices.</li>
 <li>Different test binaries for each role.</li>
 </ul>
 </li>
 </ul>

 <h2 id="example-multi-device">Example: Multi-device test plan</h2>
 <p>This example specifies the config for two devices:</p>
 <ul>
 <li>Device 1 includes a build provider and
 <code>VtsDeviceInfoCollector</code> target preparer.</li>
 <li>Device 2 includes an additional <code>FilePusher</code> preparer that pushes
 a group of host-driven related files to the device.</li>
 </ul>

 <pre class="devsite-click-to-copy">
 &lt;configuration description="VTS Codelab Plan"&gt;
   ...
 &lt;device name="device1"&gt;
 &lt;build_provider class="com.android.compatibility.common.tradefed.build.CompatibilityBuildProvider" /&gt;
 &lt;target_preparer class="com.android.tradefed.targetprep.VtsDeviceInfoCollector" /&gt;
 &lt;/device&gt;
 &lt;device name="device2" &gt;
 &lt;build_provider class="com.android.compatibility.common.tradefed.build.CompatibilityBuildProvider" /&gt;
 &lt;target_preparer class="com.android.tradefed.targetprep.VtsDeviceInfoCollector" /&gt;
 &lt;target_preparer class="com.android.compatibility.common.tradefed.targetprep.VtsFilePusher"&gt;
 &lt;option name="push-group" value="HostDrivenTest.push" /&gt;
 &lt;/target_preparer&gt;
 &lt;/device&gt;
 &lt;option name="compatibility:include-filter" value="VtsCodelabHelloWorldMultiDeviceTest" /&gt;
 &lt;/configuration&gt;
 </pre>

 <h2 id="example-hostside">Example: Host-side Python test script</h2>
 <p>For details and examples on test preparers, see
 <a href="#Test-preparers">Test Preparers</a>. For a complete host-side
 multi-device example, refer to the
 <a href="https://android.googlesource.com/platform/test/vts/+/master/testcases/codelab/hello_world_multi" class="external">hello_world_multi
 codelab</a>.
 </p>

 <pre class="devsite-click-to-copy">
 def setUpClass(self):
 logging.info('number of device: %s', self.android_devices)
 asserts.assertEqual(len(self.android_devices), 2, 'number of device is wrong.')
 self.dut1 = self.android_devices[0]
 self.dut2 = self.android_devices[1]
 self.shell1 = self.dut1.shell
 self.shell2 = self.dut2.shell

 def testSerialNotEqual(self):
 '''Checks serial number from two device not being equal.'''
 command = 'getprop | grep ro.serial'
 res1 = self.shell1.Execute(command)
 res2 = self.shell2.Execute(command)

 def getSerialFromShellOutput(output):
 '''Get serial from getprop query'''
 return output[const.STDOUT][0].strip().split(' ')[-1][1:-1]
 serial1 = getSerialFromShellOutput(res1)
 serial2 = getSerialFromShellOutput(res2)

 logging.info('Serial number of device 1 shell output: %s', serial1)
 logging.info('Serial number of device 2 shell output: %s', serial2)
 asserts.assertNotEqual(serial1, serial2, 'serials from two devices should not be the same')
 asserts.assertEqual(serial1, self.dut1.serial, 'serial got from device system property is different from allocated serial')
 asserts.assertEqual(serial2, self.dut2.serial, 'serial got from device system property is different from allocated serial')
 </pre>

 </body>
 </html>
	<html devsite>
	<head>
	<title>Multi-Device Testing</title>
	<meta name="project_path" value="/_project.yaml" />
	<meta name="book_path" value="/_book.yaml" />
	</head>
	<body>
	<!--
	Copyright 2018 The Android Open Source Project

	Licensed under the Apache License, Version 2.0 (the "License");
	you may not use this file except in compliance with the License.
	You may obtain a copy of the License at

	http://www.apache.org/licenses/LICENSE-2.0

	Unless required by applicable law or agreed to in writing, software
	distributed under the License is distributed on an "AS IS" BASIS,
	WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	See the License for the specific language governing permissions and
	limitations under the License.
	-->


	<p>VTS supports tests that require interaction between multiple Android
	devices.</p>

	<h2 id="architecture">Architecture</h2>
	<p>VTS uses the TradeFed framework to get and pass device serials to test
	modules.</p>

	<p><img src="images/vts_device_serials.png"></p>
	<figcaption><strong>Figure 1.</strong> VTS passing device serials.</figcaption>

	<p>Device requirements, such as number of devices and device types, are
	specified in test plan configuration. For example, you can specify a test plan
	that requires two Android devices with Sailfish build targets.</p>

	<h3 id="device-allocation">Device allocation</h3>
	<p>The test infrastructure (usually the test scheduler) allocates available
	devices that satisfy the requirements specified in test plan configuration to
	the VTS framework. Allocated devices are reserved for the test plan even if the
	test module is not using them. VTS agent binaries are then pushed to and run on
	all allocated devices (unless specifically instructed not to run). This ensures
	that TCP connections for shell commands and HAL RPCs are available for all
	devices in a test script.</p>

	<h3 id="test-preparers">Test preparers</h3>
	<p>The framework runs test preparers for all devices for which it received
	serial numbers. Target preparers can be single or multi-device:</p>
	<ul>
	<li>Single-device target preparers (example at
	<a href="https://android.googlesource.com/platform/test/vts/+/master/harnesses/tradefed/src/com/android/tradefed/targetprep/VtsDeviceInfoCollector.java" class="external">VtsDeviceInfoCollector</a>):
	<ul>
	<li>Can be specified only in test plan configuration with the required
	device list (future versions will allow module level configuration).</li>
	<li>Receive only one device serial.</li>
	<li>Run preparing and cleanup tasks against a specific device.</li>
	</ul>
	</li>
	<li>Multi-device target preparers (example at
	<a href="https://android.googlesource.com/platform/test/vts/+/master/harnesses/tradefed/src/com/android/tradefed/targetprep/VtsPythonVirtualenvPreparer.java" class="external">VtsPythonVirtualenvPreparer</a>):
	<ul>
	<li>Can be specified in test plan configuration or test module
	configuration</li>
	<li>Receive all device serials</li>
	<li>Run preparing and cleanup tasks for each device or all devices.</li>
	</ul>
	</li>
	</ul>

	<h3 id="test-modules">Test modules</h3>
	<p>Test modules get a list of devices after the test preparers finish setting up
	the host/devices. One host-side Python test module runs for each multi-device
	test module. Allocated Android devices are accessible from Python test modules
	as a list of
	<a href="https://android.googlesource.com/platform/test/vts/+/master/utils/python/controllers/android_device.py#322" class="external">AndroidDevice</a>
	objects:</p>
	<pre class="devsite-click-to-copy">
	devices = self.android_devices
	device1 = devices[0]
	device1_serial = device1.serial
	</pre>

	<p>All allocated devices are reserved for the test plan, even though a test
	module in the plan is only using one device.</p>

	<h2 id="device-communication">Device communication during testing</h1>
	<p>Effective multi-Android tests involve communication between allocated
	devices. When developing such tests, you must determine how to establish
	communication between the allocated devices. The following sections provide
	three communication examples (however, test developers are free to design other
	models).</p>

	<h3 id="type1">Type 1: Host-side HAL tests</h3>
	<p>Host-side HAL tests can use VTS HAL drivers that are pushed to devices by
	default:</p>

	<p><img src="images/vts_hostside_hal.png"></p>
	<figcaption><strong>Figure 2.</strong> Host-side HAL test.</figcaption>

	<p>In this scenario:</p>
	<ul>
	<li>Test logic executes on the host.</li>
	<li>Host-side test script issues RPC calls to the drivers on each device.</li>
	<li>Host side coordinates device interactions.</li>
	</ul>

	<h3 id="type2">Type 2: Host-side agent-based tests</h3>
	<p>Instead of using VTS agents on device, a host-side test can also push its own
	agent (app or binary) to each device:</p>

	<p><img src="images/vts_hostside_agent.png"></p>
	<figcaption><strong>Figure 3.</strong> Host-side, agent-based test.</figcaption>

	<p>In this scenario:</p>
	<ul>
	<li>Test logic executes on the host.</li>
	<li>Agent app (or binary) installs on each device.</li>
	<li>Host-side test script issues commands to apps on each device.</li>
	<li>Host side coordinates device interactions.</li>
	</ul>

	<p>For example, the
	<a href="https://android.googlesource.com/platform/test/vts-testcase/nbu/+/master" class="external">Next
	Billion User tests</a> in current VTS repo are host-side, app-based,
	multi-device tests.</p>

	<h3 id="type3">Type 3: Target-side HIDL tests</h2>
	<p>Target-side, multi-device HIDL tests put all test logic on device-side test
	binaries, which requires the tests to synchronize devices during test
	execution:</p>

	<p><img src="images/vts_target_hidl.png"></p>
	<figcaption><strong>Figure 4. </strong>Target-based HIDL test.</figcaption>

	<p>In this scenario:</p>
	<ul>
	<li>Test logic executes on devices.</li>
	<li>Host-side framework provides initial device identification.</li>
	<li>Target-side test binary requires synchronization:<ul>
	<li>Same test binary for all devices.</li>
	<li>Different test binaries for each role.</li>
	</ul>
	</li>
	</ul>

	<h2 id="example-multi-device">Example: Multi-device test plan</h2>
	<p>This example specifies the config for two devices:</p>
	<ul>
	<li>Device 1 includes a build provider and
	<code>VtsDeviceInfoCollector</code> target preparer.</li>
	<li>Device 2 includes an additional <code>FilePusher</code> preparer that pushes
	a group of host-driven related files to the device.</li>
	</ul>

	<pre class="devsite-click-to-copy">
	<configuration description="VTS Codelab Plan">
	...
	<device name="device1">
	<build_provider class="com.android.compatibility.common.tradefed.build.CompatibilityBuildProvider" />
	<target_preparer class="com.android.tradefed.targetprep.VtsDeviceInfoCollector" />
	</device>
	<device name="device2" >
	<build_provider class="com.android.compatibility.common.tradefed.build.CompatibilityBuildProvider" />
	<target_preparer class="com.android.tradefed.targetprep.VtsDeviceInfoCollector" />
	<target_preparer class="com.android.compatibility.common.tradefed.targetprep.VtsFilePusher">
	<option name="push-group" value="HostDrivenTest.push" />
	</target_preparer>
	</device>
	<option name="compatibility:include-filter" value="VtsCodelabHelloWorldMultiDeviceTest" />
	</configuration>
	</pre>

	<h2 id="example-hostside">Example: Host-side Python test script</h2>
	<p>For details and examples on test preparers, see
	<a href="#Test-preparers">Test Preparers</a>. For a complete host-side
	multi-device example, refer to the
	<a href="https://android.googlesource.com/platform/test/vts/+/master/testcases/codelab/hello_world_multi" class="external">hello_world_multi
	codelab</a>.
	</p>

	<pre class="devsite-click-to-copy">
	def setUpClass(self):
	logging.info('number of device: %s', self.android_devices)
	asserts.assertEqual(len(self.android_devices), 2, 'number of device is wrong.')
	self.dut1 = self.android_devices[0]
	self.dut2 = self.android_devices[1]
	self.shell1 = self.dut1.shell
	self.shell2 = self.dut2.shell

	def testSerialNotEqual(self):
	'''Checks serial number from two device not being equal.'''
	command = 'getprop \| grep ro.serial'
	res1 = self.shell1.Execute(command)
	res2 = self.shell2.Execute(command)

	def getSerialFromShellOutput(output):
	'''Get serial from getprop query'''
	return output[const.STDOUT][0].strip().split(' ')[-1][1:-1]
	serial1 = getSerialFromShellOutput(res1)
	serial2 = getSerialFromShellOutput(res2)

	logging.info('Serial number of device 1 shell output: %s', serial1)
	logging.info('Serial number of device 2 shell output: %s', serial2)
	asserts.assertNotEqual(serial1, serial2, 'serials from two devices should not be the same')
	asserts.assertEqual(serial1, self.dut1.serial, 'serial got from device system property is different from allocated serial')
	asserts.assertEqual(serial2, self.dut2.serial, 'serial got from device system property is different from allocated serial')
	</pre>

	</body>
	</html>