autoupdate_EndToEndTest: Reinstall stateful if DUT was powerwashed when installing source

When installing an old image, the DUT may be powerwashed: it happens
when EXT4 crypto is enabled but an older image can not handle it.

When it happens, reinstalled stateful.tgz to be able to continue the
test. The signal used for powerwash is the presence of rsync, only
installed in /mnt/stateful.

BUG=chromium:689105
TEST=Enable encrypt on a samus (tune2fs -O encrypt /dev/sda1)
Do an end to end test to FSI R41 6680.57.0 and back to R59 latest.

Change-Id: Iaea3691bee0ee52456e55f2876d55dcb055d8359
Signed-off-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/459125
Reviewed-by: Xixuan Wu <xixuan@chromium.org>
diff --git a/server/hosts/abstract_ssh.py b/server/hosts/abstract_ssh.py
index d2fa8ba..be281e8 100644
--- a/server/hosts/abstract_ssh.py
+++ b/server/hosts/abstract_ssh.py
@@ -114,14 +114,14 @@
 
         # Check if rsync is available on the remote host. If it's not,
         # don't try to use it for any future file transfers.
-        self._use_rsync = self._check_rsync()
+        self._use_rsync = self.check_rsync()
         if not self._use_rsync:
             logging.warning("rsync not available on remote host %s -- disabled",
                          self.hostname)
         return self._use_rsync
 
 
-    def _check_rsync(self):
+    def check_rsync(self):
         """
         Check if rsync is available on the remote host.
         """
diff --git a/server/site_tests/autoupdate_EndToEndTest/autoupdate_EndToEndTest.py b/server/site_tests/autoupdate_EndToEndTest/autoupdate_EndToEndTest.py
index b9a1592..6c9a81d 100755
--- a/server/site_tests/autoupdate_EndToEndTest/autoupdate_EndToEndTest.py
+++ b/server/site_tests/autoupdate_EndToEndTest/autoupdate_EndToEndTest.py
@@ -955,14 +955,13 @@
         @param stateful_url: If set, the specified url to find the stateful
                              payload.
         """
+        # Reboot to get us into a clean state.
+        self._host.reboot()
         try:
-            # Reboot to get us into a clean state.
-            self._host.reboot()
             # Since we are installing the source image of the test, clobber
             # stateful.
             self._update_via_test_payloads(devserver_hostname, image_url,
                                            stateful_url, True)
-            self._host.reboot()
         except OmahaDevserverFailedToStart as e:
             logging.fatal('Failed to start private devserver for installing '
                           'the source image (%s) on the DUT', image_url)
@@ -970,10 +969,19 @@
                     'Failed to start private devserver for installing the '
                     'source image on the DUT: %s' % e)
         except error.AutoservRunError as e:
-            logging.fatal('Error re-imaging or rebooting the DUT with the '
-                          'source image from %s', image_url)
-            raise error.TestError('Failed to install the source image or '
-                                  'reboot the DUT: %s' % e)
+            logging.fatal('Error re-imaging the DUT with '
+                          'the source image from %s', image_url)
+            raise error.TestError('Failed to install '
+                          'the source image DUT: %s' % e)
+        self._host.reboot()
+
+        # If powerwashed, need to reinstall stateful_url
+        if not self._host.check_rsync():
+            logging.warn('Device has been powerwashed, need to reinstall '
+                         'stateful from %s', stateful_url)
+            self._update_via_test_payloads(devserver_hostname, None,
+                                           stateful_url, True)
+            self._host.reboot()
 
 
     def _stage_artifacts_onto_devserver(self, test_conf):