blob: d7bb0f179f25953779fc93b029c5dead8b9ec62d [file] [log] [blame]
<meta charset="utf-8" lang="kotlin">
# Lint Check Unit Testing
Lint has a dedicated testing library for lint checks. To use it,
add this dependency to your lint check Gradle project:
```
testImplementation "com.android.tools.lint:lint-tests:$lintVersion"
```
This lends itself nicely to test-driven development. When we get bug
reports of a false positive, we typically start by adding the text for
the repro case, ensure that the test is failing, and then work on the
bug fix (often setting breakpoints and debugging through the unit test)
until it passes.
## Creating a Unit Test
Here's a sample lint unit test for a simple, sample lint check which
just issues warnings whenever it sees the word “lint” mentioned
in a string:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~kotlin linenumbers
package com.example.lint.checks
import com.android.tools.lint.checks.infrastructure.TestFiles.java
import com.android.tools.lint.checks.infrastructure.TestLintTask.lint
import org.junit.Test
class SampleCodeDetectorTest {
@Test
fun testBasic() {
lint().files(
java(
"""
package test.pkg;
public class TestClass1 {
// In a comment, mentioning "lint" has no effect
private static String s1 = "Ignore non-word usages: linting";
private static String s2 = "Let's say it: lint";
}
"""
).indented()
)
.issues(SampleCodeDetector.ISSUE)
.run()
.expect(
"""
src/test/pkg/TestClass1.java:5: Warning: This code mentions lint: Congratulations [ShortUniqueId]
private static String s2 = "Let's say it: lint";
∼∼∼∼∼∼∼∼∼∼∼∼∼∼∼∼∼∼∼∼
0 errors, 1 warnings
"""
)
}
}
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Lint's testing API is a “fluent API”; you chain method calls together,
and the return objects determine what is allowed next.
Notice how we construct a test object here on line 10 with the `lint()`
call. This is a “lint test task”, which has a number of setup methods
on it (such as the set of source files we want to analyze), the issues
it should consider, etc.
Then, on line 23, the `run()` method. This runs the lint unit test, and
then it returns a result object. On the result object we have a number
of methods to verify that the test succeeded. For a test making sure we
don't have false positives, you can just call `expectClean()`. But the
most common operation is to call `expect(output)`.
!!! Tip
Notice how we're including the whole text output here; including not
just the error message and line number, but lint's output of the
relevant line and the error range (using ~~~~ characters).
This is the recommended practice for lint checks. It may be tempting
to avoid “duplication” of repeating error messages in the tests
(“DRY”), so some developers have written tests where they just
assert that a given test has say “2 warnings”. But this isn't
testing that the error range is exactly what you expect (which
matters a lot when users are seeing the lint check from the IDE,
since that's the underlined region), and it could also continue to
pass even if the errors flagged are no longer what you intended.
Finally, even if the location is correct today, it may not be
correct tomorrow. Several times in the past, some unit tests in
lint's built-in checks have started failing after an update to the
Kotlin compiler because of some changes to the AST which required
tweaks here and there.
## Computing the Expected Output
You may wonder how we knew what to paste into our `expect` call
to begin with.
We didn't. When you write a test, simply start with
`expect("")`, and run the test. It will fail. You can now
copy the actual output into the `expect` call as the expected
output, provided of course that it's correct!
## Test Files
On line 11, we construct a Java test file. We call `java(...)` and pass
in the source file contents. This constructs a `TestFile`, and there
are a number of different types of test source files, such as for
Kotlin files, manifest files, icons, property files, and so on.
Using test file descriptors like this has a number of advantages
over the traditional approach of checking in test files as sources:
* Everything is kept together, so it's easier to look at a test and see
what it analyzes and what the expected results are. This is
particularly important for complex lint checks which test a lot of
scenarios. As of this writing, `ApiDetectorTest` has 157 individual
unit tests.
* Lint can provide a DSL to construct test files easily. For example,
`projectProperties().compileSdk(17)` and
`manifest().minSdk(5).targetSdk(17)` construct a `project.properties`
and an `AndroidManifest.xml` file with the correct contents to
specify for example the right <uses-sdk> element setting up the
`minSdkVersion` and `targetSdkVersion`.
For icons, we can construct bitmaps like this:
```
image("res/mipmap-hdpi/my_launcher2_round.png", 50, 50)
.fillOval(0, 0, 50, 50, 0xFFFFFFFF)
.text(5, 5, "x", 0xFFFFFFFF))
```
* Similarly, when we construct `java()` or `kotlin()` test sources, we
don't have to name the files, because lint will analyze the source
code and figure out what the class file should be named and where to
place it.
* We can easily “parameterize” our test files. For example, if you
want to run your unit test against a 100K json file, you can
construct it programmatically; you don't have to check one in.
* Since test sources often (deliberately!) have errors in them, this
sometimes causes problems with the tooling; for example, some code
review tools will flag “disallowed” constructs or things like tabs or
trailing spaces, which may be deliberate in a lint unit test.
* Lint originally checked in test sources as individual files.
Unfortunately over time, source files ended up getting reused by
multiple tests. And that made it harder to make changes, or figure
out whether test sources are still in use, and so on.
* Last but not least, because all the test construction methods
specify the correct mime type for their string parameters, IntelliJ
will actually syntax highlight the test source declarations! Here's
how this looks:
![Screenshot of nested highlighting](nested-syntax-highlighting.png)
## Trimming indents?
Notice how in the above Kotlin unit tests we used raw strings, **and**
we indented the sources to be flush with the opening “”“ string
delimiter.
You might be tempted to call `.trimIndent()` on the raw string.
However, doing that would break the above nested syntax highlighting
method (or at least it used to). Therefore, instead, call `.indented()`
on the test file itself, not the string, as shown on line 20.
Note that we don't need to do anything with the `expect` call; lint
will automatically call `trimIndent()` on the string passed in to it.
## Dollars in Raw Strings
Kotlin requires that raw strings have to escape the dollar ($)
character. That's normally not a problem, but for some source files, it
makes the source code look **really** messy and unreadable.
For that reason, lint will actually convert $ into $ (a unicode wide
dollar sign). Lint lets you use this character in test sources, and it
always converts the test output to use it (though it will convert in
the opposite direction when creating the test sources on disk).
## Quickfixes
If your lint check registers quickfixes with the reported incidents,
it's trivial to test these as well.
For example, for a lint check result which flags two incidents, with a
single quickfix, the unit test looks like this:
```
lint().files(...)
.run()
.expect(expected)
.expectFixDiffs(
""
+ "Fix for res/layout/textsize.xml line 10: Replace with sp:\n"
+ "@@ -11 +11\n"
+ "- android:textSize=\"14dp\" />\n"
+ "+ android:textSize=\"14sp\" />\n"
+ "Fix for res/layout/textsize.xml line 15: Replace with sp:\n"
+ "@@ -16 +16\n"
+ "- android:textSize=\"14dip\" />\n"
+ "+ android:textSize=\"14sp\" />\n");
```
The `expectFixDiffs` method will iterate over all the incidents it
found, and in succession, apply the fix, diff the two sources, and
append this diff along with the fix message into the log.
When there are multiple fixes offered for a single incident, it will
iterate through all of these too:
```
lint().files(...)
.run()
.expect(expected)
.expectFixDiffs(
+ "Fix for res/layout/autofill.xml line 7: Set autofillHints:\n"
+ "@@ -12 +12\n"
+ " android:layout_width=\"match_parent\"\n"
+ " android:layout_height=\"wrap_content\"\n"
+ "+ android:autofillHints=\"|\"\n"
+ " android:hint=\"hint\"\n"
+ " android:inputType=\"password\" >\n"
+ "Fix for res/layout/autofill.xml line 7: Set importantForAutofill=\"no\":\n"
+ "@@ -13 +13\n"
+ " android:layout_height=\"wrap_content\"\n"
+ " android:hint=\"hint\"\n"
+ "+ android:importantForAutofill=\"no\"\n"
+ " android:inputType=\"password\" >\n"
+ " \n");
```
## Library Dependencies and Stubs
Let's say you're writing a lint check for something like the Android
Jetpack library's `RecyclerView` widget.
In this case, it's highly likely that your unit test will reference
`RecyclerView`. But how does lint know what `RecyclerView` is? If it
doesn't, type resolve won't work, and as a result the detector won't.
You could make your test ”depend“ on the `RecyclerView`. This is
possible, using the `LibraryReferenceTestFile`, but is not recommended.
Instead, the recommended approach is to just use ”stubs“; create
skeleton classes which represent only the **signatures** of the
library, and in particular, only the subset that your lint check cares
about.
For example, for lint's own `RecyclerView` test, the unit test declares
a field holding the recycler view stub:
```
private val recyclerViewStub = java(
"""
package android.support.v7.widget;
import android.content.Context;
import android.util.AttributeSet;
import android.view.View;
import java.util.List;
// Just a stub for lint unit tests
public class RecyclerView extends View {
public RecyclerView(Context context, AttributeSet attrs) {
super(context, attrs);
}
public abstract static class ViewHolder {
public ViewHolder(View itemView) {
}
}
public abstract static class Adapter<VH extends ViewHolder> {
public abstract void onBindViewHolder(VH holder, int position);
public void onBindViewHolder(VH holder, int position, List<Object> payloads) {
}
public void notifyDataSetChanged() { }
}
}
"""
).indented()
```
And now, all the other unit tests simply include `recyclerViewStub`
as one of the test files. For a larger example, see
[this test](https://cs.android.com/android-studio/platform/tools/base/+/mirror-goog-studio-master-dev:lint/libs/lint-tests/src/test/java/com/android/tools/lint/checks/SliceDetectorTest.kt).
## Binary and Compiled Source Files
If you need to use binaries in your unit tests, there is
a special test file type for that: base64gzip. Here's an
example from a lint check which tries to recognize usage
of Cordova in the bytecode:
```
fun testVulnerableCordovaVersionInClasses() {
lint().files(
base64gzip(
"bin/classes/org/apache/cordova/Device.class",
"" +
"yv66vgAAADIAFAoABQAPCAAQCQAEABEHABIHABMBAA5jb3Jkb3ZhVmVyc2lv" +
"bgEAEkxqYXZhL2xhbmcvU3RyaW5nOwEABjxpbml0PgEAAygpVgEABENvZGUB" +
"AA9MaW5lTnVtYmVyVGFibGUBAAg8Y2xpbml0PgEAClNvdXJjZUZpbGUBAAtE" +
"ZXZpY2UuamF2YQwACAAJAQAFMi43LjAMAAYABwEAGW9yZy9hcGFjaGUvY29y" +
"ZG92YS9EZXZpY2UBABBqYXZhL2xhbmcvT2JqZWN0ACEABAAFAAAAAQAJAAYA" +
"BwAAAAIAAQAIAAkAAQAKAAAAHQABAAEAAAAFKrcAAbEAAAABAAsAAAAGAAEA" +
"AAAEAAgADAAJAAEACgAAAB4AAQAAAAAABhICswADsQAAAAEACwAAAAYAAQAA" +
"AAUAAQANAAAAAgAO"
)`
).run().expect(
```
Here, ”base64gzip“ means that the file is gzipped and then base64
encoded.
If you want to compute the base64gzip string for a given file, a simple
way to do it is to add this statement at the beginning of your test:
```
assertEquals("", TestFiles.toBase64gzip(File("/tmp/mybinary.bin")))
```
The test will fail, and now you have your output to copy/paste into the
test.
However, if you're writing byte-code based tests, don't just hard code
in the .class file or .jar file contents like this. Lint's own unit
tests did that, and it's hard to later reconstruct what the byte code
was later if you need to make changes or extend it to other bytecode
formats.
Instead, use the new `compiled` or `bytecode` test files. The key here
is that they automate a bit of the above process: the test file
provides a source test file, as well as a set of corresponding binary
files (since a single source file can create multiple class files, and
for Kotlin, some META-INF data).
Initially, you just specify the sources, and when no binary data
has been provided, lint will instead attempt to compile the sources
and emit the full test file registration.
This isn't just a convenience; lint's test infrastructure also uses
this to test some additional scenarios (for example, in a multi- module
project it will only provide the binaries, not the sources, for
upstream modules.)
## My Detector Isn't Invoked From a Test!
One common question we hear is
> My Detector works fine when I run it in the IDE or from Gradle, but
> from my unit test, my detector is never called! Why?
This is almost always because the test sources are referring to some
library or dependency which isn't on the class path. See the ”Library
Dependencies and Stubs“ section above, as well as the [frequently asked
questions](faq.md.html).
<!-- Markdeep: --><style class="fallback">body{visibility:hidden;white-space:pre;font-family:monospace}</style><script src="markdeep.min.js" charset="utf-8"></script><script src="https://morgan3d.github.io/markdeep/latest/markdeep.min.js" charset="utf-8"></script><script>window.alreadyProcessedMarkdeep||(document.body.style.visibility="visible")</script>