libraries/flicker/README.md - platform/platform_testing - Git at Google

 # Flicker Test Library

 ## Motivation
 Detect *flicker* &mdash; any discontinuous, or unpredictable behavior seen during UI transitions that is not due to performance.
 This is often the result of a logic error in the code and difficult to identify because the issue is transient and at times difficult to reproduce.
 This library helps create integration tests between `SurfaceFlinger`, `WindowManager` and `SystemUI` to identify flicker.

 Examples of flickers are:
 * Elements drawn in the wrong location (e.g., not rotating correctly)
 * Empty areas on the screen (e.g., not all areas of the screen covered during rotation)
 * System elements not appearing (e.g., nav and status bar or split screen divider)
 * Elements (dis)appearing at the wrong time (e.g., an element becomes invisible before being completely occluded)

 ## Usage and Capabilities

 The library builds and runs UI transitions, captures Winscope [traces](https://source.android.com/devices/graphics/tracing-win-transitions) and exposes common assertions that can be tested against each trace.

 ## Building a transition

 Start by defining common or error prone transitions using `TransitionRunner`.
 ```java
 // Example: Build a transition that cold launches an app from launcher
 TransitionRunner transition = new TransitionBuilder()
                 // Specify a tag to identify the transition (optional)
                 .withTag("OpenAppCold_" + testApp.getLauncherName())

                 // Specify preconditions to setup the device
                 // Wake up device and go to home screen
                 .runBeforeAll(AutomationUtils::wakeUpAndGoToHomeScreen)

                 // Setup transition under test
                 // Press the home button and close the app to test a cold start
                 .runBefore(device::pressHome)
                 .runBefore(testApp::exit)

                 // Run the transition under test
                 // Open the app and wait for UI to be idle
                 // This is the part of the transition that will be tested.
                 .run(testApp::open)
                 .run(device::waitForIdle)

                 // Perform any tear downs
                 // Close the app
                 .runAfterAll(testApp::exit)

                 // Number of times to repeat the transition to catch any flaky issues
                 .repeat(5);
 ```

 Run the transition to get a list of `TransitionResult` for each time the transition is repeated.
 ```java
     List<TransitionResult> results = transition.run();
 ```
 `TransitionResult` contains paths to test artifacts such as Winscope traces and screen recordings.

 ### Checking Assertions
 Each `TransitionResult` can be tested using an extension of the Google Truth library, `LayersTraceSubject` and `WmTraceSubject`.
 They try to balance test principles set out by Google Truth (not supporting nested assertions, keeping assertions simple) with providing support for common assertion use cases.

 Each trace can be represented as a ordered collection of trace entries, with an associated timestamp.
 Each trace entry has common assertion checks.
 The trace subjects expose methods to filter the range of entries and test for changing assertions.

 ```java
     TransitionResult result = results.get(0);
     Rect displayBounds = getDisplayBounds();

     // check all trace entries
     assertThat(result).coversRegion(displayBounds).forAllEntries();

     // check a range of entries
     assertThat(result).coversRegion(displayBounds).forRange(startTime, endTime);

     // check first entry
     assertThat(result).coversRegion(displayBounds).inTheBeginning();

     // check last entry
     assertThat(result).coversRegion(displayBounds).atTheEnd();

     // check a change in assertions, e.g. wallpaper window is visible,
     // then wallpaper window becomes and stays invisible
     assertThat(result)
                 .showsBelowAppWindow("wallpaper")
                 .then()
                 .hidesBelowAppWindow("wallpaper")
                 .forAllEntries();
 ```

 All assertions return `Result` which contains a `success` flag, `assertionName` string identifier, and `reason` string to provide actionable details to the user.
 The `reason` string is build along the way with all the details as to why the assertions failed and any hints which might help the user determine the root cause.
 Failed assertion message will also contain a path to the trace that was tested.
 Example of a failed test:

 ```
     java.lang.AssertionError: Not true that <com.android.server.wm.traces.common.LayersTrace@65da4cc>
     Layers Trace can be found in: /layers_trace_emptyregion.pb
     Timestamp: 2308008331271
     Assertion: coversRegion
     Reason:   Region to test: Rect(0, 0 - 1440, 2880)
     first empty point: 0, 99
     visible regions:
     StatusBar#0Rect(0, 0 - 1440, 98)
     NavigationBar#0Rect(0, 2712 - 1440, 2880)
     ScreenDecorOverlay#0Rect(0, 0 - 1440, 91)
     ...
         at com.google.common.truth.FailureStrategy.fail(FailureStrategy.java:24)
         ...
 ```

 ---

 ## Running Tests

 The tests can be run as any other Android JUnit tests. `frameworks/base/tests/FlickerTests` uses the library to test common UI transitions. Run `atest FlickerTests` to execute these tests.

 ---

 ## Other Topics
 ### Monitors
 Monitors capture test artifacts for each transition run. They are started before each iteration of the test transition (after the `runBefore` calls) and stopped after the transition is completed. Each iteration will produce a new test artifact. The following monitors are available:

 #### LayersTraceMonitor
 Captures Layers trace. This monitor is started by default. Build a transition with `skipLayersTrace()` to disable this monitor.
 #### WindowManagerTraceMonitor
 Captures Window Manager trace. This monitor is started by default. Build a transition with `skipWindowManagerTrace()` to disable this monitor.
 #### ScreenRecorder
 Captures screen to a video file. This monitor is disabled by default. Build a transition with `recordEachRun()` to capture each transition or build with `recordAllRuns()` to capture every transition including setup and teardown.

 ---

 ### Extending Assertions

 To add a new assertion, add a function to one of the trace entry classes, `LayersTrace.Entry` or `WindowManagerTrace.Entry`.

 ```java
     // Example adds an assertion to the check if layer is hidden by parent.
     Result isHiddenByParent(String layerName) {
         // Result should contain a details if assertion fails for any reason
         // such as if layer is not found or layer is not hidden by parent
         // or layer has no parent.
         // ...
     }
 ```
 Then add a function to the trace subject `LayersTraceSubject` or `WmTraceSubject` which will add the assertion for testing. When the assertion is evaluated, the trace will first be filtered then the assertion will be applied to the remaining entries.

 ```java
     public LayersTraceSubject isHiddenByParent(String layerName) {
         mChecker.add(entry -> entry.isHiddenByParent(layerName),
                 "isHiddenByParent(" + layerName + ")");
         return this;
     }
 ```

 To use the new assertion:
 ```java
     // Check if "Chrome" layer is hidden by parent in the first trace entry.
     assertThat(result).isHiddenByParent("Chrome").inTheBeginning();
 ```
	# Flicker Test Library

	## Motivation
	Detect flicker — any discontinuous, or unpredictable behavior seen during UI transitions that is not due to performance.
	This is often the result of a logic error in the code and difficult to identify because the issue is transient and at times difficult to reproduce.
	This library helps create integration tests between `SurfaceFlinger`, `WindowManager` and `SystemUI` to identify flicker.

	Examples of flickers are:
	* Elements drawn in the wrong location (e.g., not rotating correctly)
	* Empty areas on the screen (e.g., not all areas of the screen covered during rotation)
	* System elements not appearing (e.g., nav and status bar or split screen divider)
	* Elements (dis)appearing at the wrong time (e.g., an element becomes invisible before being completely occluded)

	## Usage and Capabilities

	The library builds and runs UI transitions, captures Winscope [traces](https://source.android.com/devices/graphics/tracing-win-transitions) and exposes common assertions that can be tested against each trace.

	## Building a transition

	Start by defining common or error prone transitions using `TransitionRunner`.
	```java
	// Example: Build a transition that cold launches an app from launcher
	TransitionRunner transition = new TransitionBuilder()
	// Specify a tag to identify the transition (optional)
	.withTag("OpenAppCold_" + testApp.getLauncherName())

	// Specify preconditions to setup the device
	// Wake up device and go to home screen
	.runBeforeAll(AutomationUtils::wakeUpAndGoToHomeScreen)

	// Setup transition under test
	// Press the home button and close the app to test a cold start
	.runBefore(device::pressHome)
	.runBefore(testApp::exit)

	// Run the transition under test
	// Open the app and wait for UI to be idle
	// This is the part of the transition that will be tested.
	.run(testApp::open)
	.run(device::waitForIdle)

	// Perform any tear downs
	// Close the app
	.runAfterAll(testApp::exit)

	// Number of times to repeat the transition to catch any flaky issues
	.repeat(5);
	```

	Run the transition to get a list of `TransitionResult` for each time the transition is repeated.
	```java
	List<TransitionResult> results = transition.run();
	```
	`TransitionResult` contains paths to test artifacts such as Winscope traces and screen recordings.

	### Checking Assertions
	Each `TransitionResult` can be tested using an extension of the Google Truth library, `LayersTraceSubject` and `WmTraceSubject`.
	They try to balance test principles set out by Google Truth (not supporting nested assertions, keeping assertions simple) with providing support for common assertion use cases.

	Each trace can be represented as a ordered collection of trace entries, with an associated timestamp.
	Each trace entry has common assertion checks.
	The trace subjects expose methods to filter the range of entries and test for changing assertions.

	```java
	TransitionResult result = results.get(0);
	Rect displayBounds = getDisplayBounds();

	// check all trace entries
	assertThat(result).coversRegion(displayBounds).forAllEntries();

	// check a range of entries
	assertThat(result).coversRegion(displayBounds).forRange(startTime, endTime);

	// check first entry
	assertThat(result).coversRegion(displayBounds).inTheBeginning();

	// check last entry
	assertThat(result).coversRegion(displayBounds).atTheEnd();

	// check a change in assertions, e.g. wallpaper window is visible,
	// then wallpaper window becomes and stays invisible
	assertThat(result)
	.showsBelowAppWindow("wallpaper")
	.then()
	.hidesBelowAppWindow("wallpaper")
	.forAllEntries();
	```

	All assertions return `Result` which contains a `success` flag, `assertionName` string identifier, and `reason` string to provide actionable details to the user.
	The `reason` string is build along the way with all the details as to why the assertions failed and any hints which might help the user determine the root cause.
	Failed assertion message will also contain a path to the trace that was tested.
	Example of a failed test:

	```
	java.lang.AssertionError: Not true that <com.android.server.wm.traces.common.LayersTrace@65da4cc>
	Layers Trace can be found in: /layers_trace_emptyregion.pb
	Timestamp: 2308008331271
	Assertion: coversRegion
	Reason: Region to test: Rect(0, 0 - 1440, 2880)
	first empty point: 0, 99
	visible regions:
	StatusBar#0Rect(0, 0 - 1440, 98)
	NavigationBar#0Rect(0, 2712 - 1440, 2880)
	ScreenDecorOverlay#0Rect(0, 0 - 1440, 91)
	...
	at com.google.common.truth.FailureStrategy.fail(FailureStrategy.java:24)
	...
	```

	---

	## Running Tests

	The tests can be run as any other Android JUnit tests. `frameworks/base/tests/FlickerTests` uses the library to test common UI transitions. Run `atest FlickerTests` to execute these tests.

	---

	## Other Topics
	### Monitors
	Monitors capture test artifacts for each transition run. They are started before each iteration of the test transition (after the `runBefore` calls) and stopped after the transition is completed. Each iteration will produce a new test artifact. The following monitors are available:

	#### LayersTraceMonitor
	Captures Layers trace. This monitor is started by default. Build a transition with `skipLayersTrace()` to disable this monitor.
	#### WindowManagerTraceMonitor
	Captures Window Manager trace. This monitor is started by default. Build a transition with `skipWindowManagerTrace()` to disable this monitor.
	#### ScreenRecorder
	Captures screen to a video file. This monitor is disabled by default. Build a transition with `recordEachRun()` to capture each transition or build with `recordAllRuns()` to capture every transition including setup and teardown.

	---

	### Extending Assertions

	To add a new assertion, add a function to one of the trace entry classes, `LayersTrace.Entry` or `WindowManagerTrace.Entry`.

	```java
	// Example adds an assertion to the check if layer is hidden by parent.
	Result isHiddenByParent(String layerName) {
	// Result should contain a details if assertion fails for any reason
	// such as if layer is not found or layer is not hidden by parent
	// or layer has no parent.
	// ...
	}
	```
	Then add a function to the trace subject `LayersTraceSubject` or `WmTraceSubject` which will add the assertion for testing. When the assertion is evaluated, the trace will first be filtered then the assertion will be applied to the remaining entries.

	```java
	public LayersTraceSubject isHiddenByParent(String layerName) {
	mChecker.add(entry -> entry.isHiddenByParent(layerName),
	"isHiddenByParent(" + layerName + ")");
	return this;
	}
	```

	To use the new assertion:
	```java
	// Check if "Chrome" layer is hidden by parent in the first trace entry.
	assertThat(result).isHiddenByParent("Chrome").inTheBeginning();
	```