When developers use a Jetpack library, it should be easy to write reliable and automated tests for their own code's functionality. In most cases, tests written against a library will need to interact with that library in some way -- setting up preconditions, reaching synchronization points, making calls -- and the library should provide necessary functionality to enable such tests, either through public APIs or optional -testing artifacts.
Testability, in this document, means how easily and effectively the users of a library can create tests for apps that use that library.
NOTE Tests that check the behavior of a library have a different mission than tests made by app developers using that library; however, library developers may find themselves in a similar situation when writing tests for sample code.
Often, the specifics of testability will vary from library to library and there is no one way of providing it. Some libraries have enough public API surface, others provide additional testing artifacts (e.g. Lifecycle Runtime Testing artifact).
The best way to check if your library is testable is to try to write a sample app with tests. Unlike regular library tests, these apps will be limited to the public API surface of the library.
Keep in mind that a good test for a sample app should have:
If you are able to write such tests for your library, you are good to go. If you struggled or found yourself writing duplicate testing code, there is room for improvement.
To get started with sample code, see Sample code in Kotlin modules for information on writing samples that can be referenced from API reference documentation or Project directory structure for module naming guidelines if you'd like to create a basic test app.
Singletons are usually bad for tests as they live across different tests, opening the gates for possible side-effects between tests. When possible, try to avoid using singletons. If it is not possible, consider providing a test artifact that will reset the state of the singleton between tests.
public class JobQueue { public static JobQueue getInstance(); }
public class JobQueue { public JobQueue(); }
object JobQueueTestUtil { fun createForTest(): JobQueue fun resetForTesting(jobQueue: JobQueue) }
Sometimes, your library might be controlling resources on the device in which case even if it is not a singleton, it might have side-effects. For instance, Room, being a database library, inherently modifies a file on the disk. To allow proper isolated testing, Room provides a builder option to create the database in memory A possible alternative solution could be to provide a test rule that will properly close databases after each test.
public class Camera { // Sends configuration to the camera hardware, which persists the // config across app restarts and applies to all camera clients. public void setConfiguration(Config c); // Retrieves the current configuration, which allows clients to // restore the camera hardware to its prior state after testing. public Config getConfiguration(); }
If your library needs an inherently singleton resource (e.g. WorkManager is a wrapper around JobScheduler and there is only 1 instance of it provided by the system), consider providing a testing artifact. To provide isolation for tests, the WorkManager library ships a separate testing artifact
A common example of this use case is libraries that do multi-threaded operations. For Kotlin libraries, this is usually achieved by receiving a coroutine context or scope. For Java libraries, it is commonly an Executor. If you have a case like this, make sure it can be passed as a parameter to your library.
NOTE Android API Guidelines require that methods accepting a callback must also take an Executor
For example, the Room library allows developers to pass different executors for background query operations. When writing a test, developers can invoke this with a custom executor where they can track work completion. See SuspendingQueryTest in Room's integration test app for implementation details.
val localDatabase = Room.inMemoryDatabaseBuilder( ApplicationProvider.getApplicationContext(), TestDatabase::class.java ) .setQueryExecutor(ArchTaskExecutor.getIOThreadExecutor()) .setTransactionExecutor(wrappedExecutor) .build() // ... wrappedExecutor.awaitTermination(1, TimeUnit.SECONDS)
If the external resource you require does not make sense as a public API, such as a main thread executor, then you can provide a testing artifact which will allow setting it. For example, the Lifecycle package depends on the main thread executor to function but for an application, customizing it does not make sense (as there is only 1 “pre-defined” main thread for an app). For testing purposes, the Lifecycle library provides a testing artifact which includes the CountingTaskExecutorRule JUnit test rule to change them.
@Rule @JvmField val countingTaskExecutorRule = CountingTaskExecutorRule() // ... @After fun teardown() { // At the end of all tests, query executor should // be idle (e.g. transaction thread released). countingTaskExecutorRule.drainTasks(500, TimeUnit.MILLISECONDS) assertThat(countingTaskExecutorRule.isIdle).isTrue() }
Sometimes, the developer might want to track side effects of your library for end-to-end testing. For instance, if your library provides some functionality that might decide to toggle Bluetooth -- outside developer‘s direct control -- it might be a good idea to have an interface for that functionality and also provide a fake that can record such calls. If you don’t think that interface makes sense as a library configuration, you can use the @RestrictTo annotation with scope LIBRARY_GROUP to restrict usage of that interface to your library group and provide a testing artifact with the fake so that developer can observe side effects only in tests while you can avoid creating unnecessary APIs.
public class EndpointConnector { public void discoverEndpoints(Executor e, Consumer<List<Endpoint>> callback); @RestrictTo(Scope.LIBRARY_GROUP) public void setBleInterface(BleInterface bleInterface); } public class EndpointConnectorTestHelper { public void setBleInterface(EndpointConnector e, BleInterface b); }
NOTE There is a fine balance between making a library configurable and making configuration a nightmare for the developer. You should try to always have defaults for these configurable objects to ensure your library is easy to use while still testable when necessary.
There are certain situations where it could be useful to provide more APIs that only make sense in the scope of testing. For example, a Lifecycle class has methods that are bound to the main thread but developers may want to have other tests that rely on lifecycle but do not run on the main thread (or even on an emulator). For such cases, you may create APIs that are testing-only to allow developers to use them only in tests while giving them the confidence that it will behave as close as possible to a real implementation. For the case above, LifecycleRegistry provides an API to create an instance of it that will not enforce thread restrictions.
NOTE Even though the implementation referenced above is acceptable, it is always better to create such functionality in additional testing artifacts when possible.
When writing Android platform APIs, testing-only APIs should be clearly distinguished from non-test API surface and restricted as necessary to prevent misuse. In some cases, the @TestApi annotation may be appropriate to restrict usage to CTS tests; however, many platform testing APIs are also useful for app developers.
class AdSelectionManager { /** * Returns testing-specific APIs for this manager. * * @throws SecurityException when called from a non-debuggable application */ public TestAdSelectionManager getTestAdSelectionManager(); }
-testing artifactIn some cases, the developer might need an instance of a class provided by your library but does not want to (or cannot) initiate it in tests. Moreover, behavior of such classes might not be fully defined for edge cases, making it difficult for developers to mock them.
For such cases, it is a good practice to provide a fake implementation out of the box that can be controlled in tests. For example, the Lifecycle library provides TestLifecycleOwner as a fake implementation for the LifecycleOwner class that can be manipulated in tests to create different use cases.
private TestLifecycleOwner mOwner = new TestLifecycleOwner( Lifecycle.State.INITIALIZED, new TestCoroutineDispatcher()); @Test public void testObserverToggle() { Observer<String> observer = (Observer<String>) mock(Observer.class); mLiveData.observe(mOwner, observer); verify(mActiveObserversChanged, never()).onCall(anyBoolean()); // ... }
Even when your library is fully testable, it is often not clear for a developer to know which APIs are safe to call in tests and when. Providing guidance on d.android.com or in a Medium post will make it much easier for the developer to start testing with your library.
Examples of testing guidance:
When writing integration tests against your code that depends on your library, look out for the following anti-patterns.
Instrumentation.waitForIdleSync() as a synchronization barrierThe waitForIdle() and waitForIdleSync() methods claim to “(Synchronously) wait for the application to be idle.” and may seem like reasonable options when there is no obvious way to observe whether an action has completed; however, these methods know nothing about the context of the test and return when the main thread's message queue is empty.
view.requestKeyboardFocus(); Instrumentation.waitForIdleSync(); sendKeyEvents(view, "1234"); // There is no guarantee that `view` has keyboard focus yet. device.pressEnter();
In apps with an active UI animation, the message queue is never empty. If the app is waiting for a callback across IPC, the message queue may be empty despite the test not reaching the desired state.
In some cases, waitForIdleSync() introduces enough of a delay that unrelated asynchronous actions happen to have completed by the time the method returns; however, this delay is purely coincidental and eventually leads to flakiness.
Instead, find a reliable synchronization barrier that guarantees the expected state has been reached or the requested action has been completed. This might mean adding listeners, callbacks, ListenableFutures, or LiveData.
See Asynchronous work in the API Guidelines for more information on exposing the state of asynchronous work to clients.
Thread.sleep() as a synchronization barrierThread.sleep() is a common source of flakiness and instability in tests. If a developer needs to call Thread.sleep() -- even indirectly via a PollingCheck -- to get their test into a suitable state for checking assertions, your library needs to provide more reliable synchronization barriers.
List<MediaItem> playlist = MediaTestUtils.createPlaylist(playlistSize); mPlayer.setPlaylist(playlist); // Wait some time for setting the playlist. Thread.sleep(TIMEOUT_MS); assertTrue(mPlayer.getPositionInPlaylist(), 0);
See the previous header for more information of providing synchronization barriers.