Incremental Processing

Incremental processing is a way to avoid re-processing of sources as much as possible. The major goal is to reduce the turn-around time of a typical change-compile-test cycle. Here is a wiki link to the more general idea of incremental computation.

To be able to know which sources are dirty, i.e., those that need to be reprocessed, KSP needs processors' help to identify the correspondence of input sources and generated outputs. Because it is cumbersome and error prone to keep track of which inputs are involved in generating which outputs, KSP is designed to help with that and only require a minimum set of sources that serve as roots that processors start to navigate the code structure. In other words, a processor needs to associate an output with sources of those KSNode, if obtained from:

  • Resolver.getAllFiles
  • Resolver.getSymbolsWithAnnotation
  • Resolver.getClassDesclarationByName

Currently, only changes in Kotlin and Java sources are tracked. If there is a change in the classpath, namely in other modules or libraries, a full re-processing will be triggered.

Incremental processing is currently enabled by default. To disable it, set the Gradle property ksp.incremental=false. To enable logs, which dump the dirty set according to dependencies and outputs, use ksp.incremental.log=true. They can be found as build/*.log.

Aggregating v.s. Isolating

The idea is similar but slightly different to the definition in Gradle annotation processing. In KSP,

  • aggregating / isolating is associated with each output, rather than the entire processor
  • an isolating output can have several sources
  • aggregating means that an output can be affected by any changes

If an output is aggregating, any changes may affect it potentially, except removal of files that don‘t affect other files. In other words, if there’s a change, all aggregating outputs need to be regenerated and therefore all of their sources will be reprocessed. Note that only registered files and changed / new files will be re-processed.

For example, an output collecting all symbols with an interesting annotation is aggregating.

If an output is not aggregating, it only depends on the sources specified. Changes in other sources do not affect it. Unlike Gradle's java annotation processing, there can be multiple source files for an output.

For example, a generated class, which is dedicated to an interface it implements, is not aggregating.

In short, if an output may depend on new or any changed sources, it is aggregating. Otherwise it is not.

For readers familiar with Java annotation processing:

  • In an isolating Java annotation processor, all the outputs are isolating in KSP.
  • In an aggregating Java annotation processor, some outputs can be isolating and some be aggregating in KSP.

Example 1

A processor generates outputForA after reading class A in A.kt and class B in B.kt, where A extends B. The processor got A by Resolver.getSymbolsWithAnnotation and then got B by KSClassDeclaration.superTypes from A. Because the inclusion of B is due to A, B.kt needn‘t to be specified in dependencies for outputForA. Note that specifying B.kt in this case doesn’t hurt, it is only unnecessary.

// A.kt
@Interesting
class A : B()

// B.kt
open class B

// Example1Processor.kt
class Example1Processor : SymbolProcessor {
    ...
    override fun process(resolver: Resolver) {
        val declA = resolver.getSymbolsWithAnnotation("Interesting").first() as KSClassDeclaration
        val declB = declA.superTypes.first().resolve().declaration
        // B.kt isn't required, because it is deducible by KSP.
        val dependencies = Dependencies(aggregating = false, declA.containingFile!!)
        // outputForA.kt
        val outputName = "outputFor${declA.simpleName.asString()}"
        // It depends on A.kt and B.kt.
        val output = codeGenerator.createNewFile(dependencies, "com.example", outputName, "kt")
        output.write("// $declA : $declB\n".toByteArray())
        output.close()
    }
    ...
}

Example 2

Consider sourceA -> outputA, sourceB -> outputB.

When sourceA is changed:

  • If outputB is aggregating
    • sourceA and sourceB are reprocessed
  • If outputB is not aggregating
    • sourceA is reprocessed.

When sourceC is added:

  • If outputB is aggregating
    • sourceC and sourceB are reprocessed
  • If outputB is not aggregating
    • sourceC is reprocessed.

When sourceA is removed:

  • nothing has to be done.

When sourceB is removed:

  • nothing has to be done.

How Dirtyness Are Determined

A dirty file is either changed by users directly, or affected by other dirty files indirectly. In KSP, propagation of dirtyness is done in 2 steps:

  • Propagation by resolution tracing: Resolving a type reference (implicitly or explicitly) is the only way to navigate from one file to another. When a type reference is resolved by a processor, a changed or affected file that contains a change that may potentially affect the resolution result will affect the file containing that reference.
  • Propagation by input-output correspondence: If a source file is changed or affected, all other source files having some output in common with that file are affected.

Note that both of them are transitive and the second forms equivalence classes.

Reporting Bugs

To report a bug, please set Gradle properties ksp.incremental=true and ksp.incremental.log=true, and start with a clean build. There are 4 log files:

  • build/kspDirtySetByDeps.log
  • build/kspDirtySetByOutputs.log
  • build/kspDirtySet.log
  • build/kspSourceToOutputs.log

They contain file names of sources and outputs, plus the timestamps of the builds. The first two are only avaiable in successive incremental builds and not available in clean builds.