This package contains nothing of note, but will eventually support extracting Kythe Compilation Units on Google Cloud Platform.
Documentation for Cloud Build itself is available at https://cloud.google.com/cloud-build/.
For the rest of this test documentation, we‘ll assume you’ve run those setup instructions. Additionally, you should make an environment variable for your gs bucket:
export BUCKET_NAME="your-bucket-name"
To make sure you have done setup correctly, we have an example binary at kythe/extractors/gcp/examples/helloworld
, which you can run as follows:
gcloud builds submit --config examples/helloworld/helloworld.yaml \ --substitutions=_BUCKET_NAME="$BUCKET_NAME"\ examples/helloworld
If that fails, you have to go back up to the Cloud Build section and follow the installation steps. Of note, you will have to install gcloud
, authorize it, associate it with a valid project id, create a test gs bucket.
To extract a maven repository using Kythe on Cloud Build, use examples/mvn.yaml
. This assumes that you will specify a maven repository in _REPO
, and that the repository has a top-level pom.xml
file (right now it is a hard-coded location, but in the future it will be configurable). This also assumes you specify $BUCKET_NAME
as per the Hello World Test above. _CORPUS
can be any identifying string for your repo, for example: “guava”.
gcloud builds submit --config examples/mvn.yaml \ --substitutions=\ _BUCKET_NAME=$BUCKET_NAME,\ _REPO=https://github.com/project-name/repo-name,\ _COMMIT=<version-hash>,\ _CORPUS=repo-name\ --no-source
To extract multiple parts of https://github.com/google/guava, use examples/guava-mvn.yaml
.
gcloud builds submit --config examples/guava-mvn.yaml \ --substitutions=\ _BUCKET_NAME=$BUCKET_NAME,\ _COMMIT=<commit-hash>,\ --no-source
This outputs guava-<commit-hash>.kzip
to $BUCKET_NAME
on Google Cloud Storage.
This is a reasonable example of a maven project which has already specified the requisite maven-compiler-plugin
bits in their pom.xml
files to support Kythe extraction, and also a project which has multiple modules.
Note however not all directories from guava extract with the top-level action. For example if you want to extract the android copy of guava that lives inside of the guava tree, you would need a slightly different action:
gcloud builds submit --config examples/guava-android-mvn.yaml \ --substitutions=\ _BUCKET_NAME=$BUCKET_NAME,\ _COMMIT=<commit-hash>\ --no-source
This outputs guava-android-<commit-hash>kzip
to $BUCKET_NAME
on GCS.
Gradle is extracted similarly:
gcloud builds submit --config examples/gradle.yaml \ --substitutions=\ _BUCKET_NAME=$BUCKET_NAME,\ _REPO=https://github.com/project-name/repo-name,\ _COMMIT=<version-hash>,\ _CORPUS=repo-name\ --no-source
_COMMIT
: git repository commit to checkout, build, and extract_REPO
: source git repository URL_BUCKET_NAME
: GCS bucket name to store extracted compilations_CORPUS
: Kythe corpus label# Extract github.com/angular/angular at commit 8accc98 gcloud builds submit --no-source --config kythe/extractors/gcp/bazel/bazel.yaml \ --substitutions=_REPO=https://github.com/angular/angular.git,\ _COMMIT=8accc98d28249628e84136d7306fdbbe1f4caaef,\ _BUCKET_NAME=$BUCKET_NAME,\ _CORPUS=github.com/angular/angular # Extract github.com/bazelbuild/bazel at commit 22d375b gcloud builds submit --no-source --config kythe/extractors/gcp/bazel/bazel.yaml \ --substitutions=_REPO=https://github.com/bazelbuild/bazel.git,\ _COMMIT=22d375bd532b04bb83f18a7770e5080e23a1d517,\ _BUCKET_NAME=$BUCKET_NAME,\ _CORPUS=github.com/bazelbuild/bazel # Extract github.com/protocolbuffers/protobuf at commit e728325 gcloud builds submit --no-source --config kythe/extractors/gcp/bazel/bazel.yaml \ --substitutions=_REPO=https://github.com/protocolbuffers/protobuf.git,\ _COMMIT=e7283254d6eb01ddfdb63cc3c89cd312e2d354d5,\ _BUCKET_NAME=$BUCKET_NAME,\ _CORPUS=github.com/protocolbuffers/protobuf
Cloud Build has a REST API described at https://cloud.google.com/cloud-build/docs/api/reference/rest/. For Kythe extraction, we have a test binary that lets you isolate authentication problems before dealing with real builds.
You will need access to your project's service credentials:
If your team already has credentials made for this purpose, see if you can re-use them.
If not, you can use these steps to create new credentials:
GOOGLE_APPLICATION_CREDENTIALS
as described in the above link.To test, run
bazel build kythe/extractors/gcp/examples/restcheck:rest_auth_check ./bazel-bin/kythe/extractors/gcp/examples/restcheck/rest_auth_check -project_id=some-project
If that returns with a 403 error, you likely did the authentication steps above incorrectly.
Kythe team maintains a few images useful for extracting Kythe data on Google Cloud Build. Many of these are used in example scripts and other generated GCB executions in Kythe.
Created from kythe/java/com/google/devtools/kythe/extractors/java/artifacts, this image contains:
javac-wrapper.sh
script which calls Kythe extraction and then an actual java compilerjavac_extractor.jar
which is the Kythe java extractorjavac9_tools.jar
which contains javac langtools for JDK 9, but targets JRE 8Created from kythe/extractors/bazel, this image contains all of the pieces of kythe necessary to extract supported languages - bazel itself, all of the kythe extractors, and the .bazelrc
. Additionally, it contains necessary tools (including a copy of kzip-tools
described below), and some required scripts.
When running this docker image, you must set environment variable $KYTHE_OUTPUT_DIRECTORY
.
This is a simple wrapper around kythe/go/extractors/config/preprocessor, which we use to preprocess build configurations to be able to specify all of the above custom javac extraction logic. Supports maven pom.xml
files and gradle build.gradle
files. Ironically, bazel extraction doesn‘t need its BUILD
files modified, because you can pass extractors directly as extra_action
, so build-preprocessor
doesn’t support BUILD
files.
This image exposes the binary kythe/go/platform/tools/kzip, which currently supports merging multiple kzips together and creating trivial kzips from the command line.
Make sure you've followed the setup setps above in Cloud Build, especially gcloud auth login
.
This, confusingly, could be two completely separate errors. First, and simpler to check, you could have just spelled the repo incorrectly. If you have a typo in the repo name, instead of telling you “repo doesn't exist” or something, the failure message is the above error about “could not read Username”.
If you have verified that the repo name is spelled correctly, then you may be trying to access a private git repo. It is possible to clone out of a private git repo, but you need to follow some extra steps. This will involve using Cloud KMS, and the steps are described in this Cloud Build Help Doc. This will involve adding extra steps to your .yaml
file for decrypting a provided key and using it to authenticate with git. Finally, your existing git clone step will need to be modified to use the same root volume as your two new steps.