ICU version upgrade

Upgrade the ICU on Android to the new upstream version. The new upstream versions can be found at https://github.com/unicode-org/icu/releases.

The below contains the steps and commands in order to upgrade the ICU version in the AOSP.

Prerequisites

Install the prerequisite tools
- See http://site.icu-project.org/repository
- git-lts has to be installed to pull binaries correctly from the github.
- git-filter-repo for rewriting the Change-Id later.
Generate a github read-access token and setup a local ~/.m2/setting.xml
- See http://cldr.unicode.org/development/maven. This is required to download the prebuilt ICU to break the circular dependency between CLDR and icu.
Check out goog/main
- http://go/repo-init/git_main

Branch Structure

external/icu project in the AOSP,
- We rebase Android-specific patches on the icu-staging branch. Then we merge into goog/main later.
external/cldr project
- We use the `goog/upstream-release-cldr as the mirror of the upstream release branch. We don’t modify this branch with Android patches, but merge this branch into goog/main where the Android patches are located.

Verifying the cleanliness

Build the clean source of AOSP
```
source build/envsetup.sh
lunch cf_x86_64_phone-trunk_staging-userdebug
m
```
The build should succeed. Otherwise, your local repo may be broken. Please consider to re-sync.

Verify that the generated artifacts are clean

If you re-run the generation scripts and it causes new changes, some patches are missing in the sources, e.g. icu4j/ and icu4c or external/cldr.
The scripts include

external/icu/tools/updatecldrdata.py
external/icu/tools/updateicudata.py
system/timezone/update-tzdata.py
external/icu/tools/srcgen/generate_android_icu4j.sh
external/icu/tools/icu4c_srcgen/generate_libandroidicu.py
external/icu/tools/icu4c_srcgen/generate_ndk.py

Steps

Configure the versions and temp directory

Customize the following environment variables. The following example targets the ICU 76.1 and CLDR 46.0 version.

export ICU_VERSION=76
export ICU_MINOR_VERSION=1
export CLDR_VERSION=46
export ICU_UPGRADE_BUG=1234567890 # buganizer bug

Set the path where you want to store the upstream ICU and CLDR sources

export UPSTREAM_CLDR_GIT=</path/to/directory/on/your/disk/cldr>
export UPSTREAM_ICU_GIT=</path/to/directory/on/your/disk/icu>

Initialize empty directories to store upstream sources if the above directories needs to be created.

mkdir -p ${UPSTREAM_CLDR_GIT}
mkdir -p ${UPSTREAM_ICU_GIT}

Copy the CLDR sources into the upstream-release-cldr branch

2a. Copy sources

export CLDR_UPSTREAM_BRANCH=release-${CLDR_VERSION}

cd ${ANDROID_BUILD_TOP}/external/cldr
git fetch goog upstream-release-cldr
git branch ${CLDR_UPSTREAM_BRANCH} --track goog/upstream-release-cldr
git checkout ${CLDR_UPSTREAM_BRANCH}

git clone https://github.com/unicode-org/cldr.git ${UPSTREAM_CLDR_GIT}

git --git-dir=${UPSTREAM_CLDR_GIT}/.git --work-tree=${UPSTREAM_CLDR_GIT} fetch
git --git-dir=${UPSTREAM_CLDR_GIT}/.git --work-tree=${UPSTREAM_CLDR_GIT} checkout ${CLDR_UPSTREAM_BRANCH}
rm -rf *
cp -r ${UPSTREAM_CLDR_GIT}/* .
git clean -dfX # Remove ignored files
git add -A
git commit -F- <<EOF
Copy upstream ${CLDR_UPSTREAM_BRANCH}

Bug: ${ICU_UPGRADE_BUG}
Test: n/a
EOF
# Upload this CL to upstream-release-cldr branch
repo upload --cbr -o nokeycheck -o uploadvalidator~skip --no-verify .

2b. Merge the upstream sources with patches in goog/main

export CLDR_BRANCH=cldr${CLDR_VERSION}-main
git branch ${CLDR_BRANCH} --track goog/main
git checkout ${CLDR_BRANCH}
git merge ${CLDR_UPSTREAM_BRANCH} -m "
Merge CLDR ${CLDR_VERSION} in upstream-release-cldr into goog/main

Bug: ${ICU_UPGRADE_BUG}
Test: external/icu/tools/updatecldrdata.py
"

2c. Resolve any merge conflicts with the Android-specific patches. Continue creating the merge commit

git merge --continue

2d. Upload the CL to main branch

repo upload --cbr -o nokeycheck -o uploadvalidator~skip --no-verify .

Note: if the remote server fails when uploading the merge commit, take a look at the workarounds in http://b/365619260

Copy ICU upstream sources into external/icu

cd ${ANDROID_BUILD_TOP}/external/icu
export ICU_BRANCH=icu-staging
export UPSTREAM_RELEASE_TAG=release-${ICU_VERSION}.${ICU_MINOR_VERSION}
git fetch goog main ${ICU_BRANCH}
git branch ${ICU_BRANCH} --track goog/${ICU_BRANCH}
git checkout ${ICU_BRANCH}
# Merge main into the staging branch.
git merge --no-ff goog/main -m
"
Merge branch goog/main into ${ICU_BRANCH}

Bug: ${ICU_UPGRADE_BUG}
Test: n/a
"
# Clone the upstream CLDR repo locally to ${UPSTREAM_ICU_GIT}
test -d ${UPSTREAM_ICU_GIT}/.git || git clone https://github.com/unicode-org/icu.git ${UPSTREAM_ICU_GIT}

git --git-dir=${UPSTREAM_ICU_GIT}/.git --work-tree=${UPSTREAM_ICU_GIT} fetch
git --git-dir=${UPSTREAM_ICU_GIT}/.git --work-tree=${UPSTREAM_ICU_GIT} checkout ${UPSTREAM_RELEASE_TAG}
find icu4j/ -type f,d ! -regex ".*/\(Android.mk\|Android.bp\|adjust_icudt_path.mk\|liblayout-jarjar-rules.txt\|.gitignore\|AndroidTest.xml\)" -delete && \
find icu4c/ -type f,d ! -regex ".*/\(Android.mk\|Android.bp\|.gitignore\|AndroidTest.xml\|robo-jarjar-rules.txt\)" -delete && \
cp -r ${UPSTREAM_ICU_GIT}/icu4j . && \
cp -r ${UPSTREAM_ICU_GIT}/icu4c . && \
git checkout HEAD -- icu4c/.gitignore icu4j/.gitignore  && # Android has extra .gitignores. Use our version. \
rm -r tools/cldr && \
cp -r ${UPSTREAM_ICU_GIT}/tools/cldr tools/cldr && \

git add -A && \
git commit -F- <<EOF
Copy ICU ${UPSTREAM_RELEASE_TAG} into goog/${ICU_BRANCH}

Copy the files with the following commands:
find icu4j/ -type f,d ! -regex ".*/\(Android.mk\|Android.bp\|adjust_icudt_path.mk\|liblayout-jarjar-rules.txt\|.gitignore\|AndroidTest.xml\)" -delete
find icu4c/ -type f,d ! -regex ".*/\(Android.mk\|Android.bp\|.gitignore\|AndroidTest.xml\|robo-jarjar-rules.txt\)" -delete
cp -r \${UPSTREAM_ICU_GIT}/icu4j .
cp -r \${UPSTREAM_ICU_GIT}/icu4c .
git checkout HEAD -- icu4c/.gitignore icu4j/.gitignore
rm -r tools/cldr
cp -r \${UPSTREAM_ICU_GIT}/tools/cldr tools/cldr
EOF

Apply Android-specific patches into external/icu
4a. Cherry-pick the patches from the last staging branch. For example using the following query for ICU 71 patches
- https://android-review.git.corp.google.com/q/%22Android+patch%22+branch:icu-staging+status:merged
- The cherry-pick command is
```
git cherry-pick <first_patch_in_the_chain>~1..<last_patch_in_the_chain>
```
- If any cherry-picks causes conflicts, please resolve them carefully.
  - If the patch and a upstream commit resolves the same issue, or the patch has been upstreamed, the patch can be discarded.
4b. Cherry-pick theq patches since the ICU upgrade
- Find the patches with this query.
  - AOSP gerrit: https://r.android.com/q/%2522Android+patch%2522+project:platform/external/icu+status:merged+-owner:automerger+-owner:android-build-coastguard-worker%2540google.com+branch:main
  - Internal gerrit: https://googleplex-android-review.git.corp.google.com/q/%2522Android+patch%2522+project:platform/external/icu+status:merged+-owner:automerger+-owner:android-build-coastguard-worker%2540google.com+branch:main
4c. Reset Change-Id in the cherry-picked CLs
- ```
THE_COPY_COMMIT=$(git log --pretty=format:'%h' -n 1 --grep "Copy ICU ${UPSTREAM_RELEASE_TAG} into goog/${ICU_BRANCH}")
git filter-branch -f --msg-filter 'sed "s/Change-Id: .*//g"' ${THE_COPY_COMMIT}..HEAD
git rebase -i ${THE_COPY_COMMIT} # It should open vim
# Replace pick with reword in vim `%s/pick /reword /g`
# Close all commit message edits with :wq in vim
# It triggers the git-hook to generate a Change-Id.
# TODO: Find a better way to generate Change-Id
```

Regenerate and commit the artifacts

5a. Update icu source data files

croot external/icu
tools/updatecldrdata.py  && \

git add -A && \
git commit  -F- <<EOF
Regenerated source data files with Android CLDR patches

Source data files updated using:
tools/updatecldrdata.py

Test: n/a
EOF

5b. Update icu binary data files

tools/updateicudata.py  && \
git add -A && \
git commit -F- <<EOF
Regenerated binary data files with Android ICU patches

Binary data files updated using:
tools/updateicudata.py

Test: n/a
EOF

5c. Pin the public API surface temporarily

We will later expose the new APIs after submitting the version upgrade CLs.

./tools/srcgen/generate_allowlisted_public_api.sh && \
git add -A && \
git commit -F- <<EOF
Pin the current API list

Test: ./tools/srcgen/generate_allowlisted_public_api.sh
EOF

5d. Regenerate android_icu4j/

Commands

tools/srcgen/generate_android_icu4j.sh && \

git add -A && \
git commit -F- <<EOF
Regenerate android_icu4j/ from icu4j/

android_icu4j files updated using:
tools/srcgen/generate_android_icu4j.sh

Test: n/a
EOF

If any java patches are not applied perfectly, remove the .orig and .rej files with the following command

find android_icu4j/ \( -name "*.orig" -o -name "*.rej" \) -delete
# Please manually update and resolve conflict of the java doc in android_icu4j/
git commit -amend
# Regenerate the patch files
./tools/srcgen/javadoc_patches/create_patches.sh
git add -A && git commit -F- <<EOF
Regenerate java doc patches

Test: n/a
EOF

5e. Re-genereate libandroidicu/ sources

./tools/icu4c_srcgen/generate_libandroidicu.py
git add -A && git commit  -F- <<EOF
Regenerate libandroidicu

The command:
./tools/icu4c_srcgen/generate_libandroidicu.py

Test: n/a
EOF

5f. Regenerate libicu/ sources

Commands

./tools/icu4c_srcgen/generate_ndk.py

git add -A && git commit -a  -F- <<EOF
Regenerate libicu.so and ICU4C CTS headers

The command:
./tools/icu4c_srcgen/generate_ndk.py

Test: n/a
EOF

Review the changes in libicu/ndk_headers/unicode. Check that no ABI change is in the C struct or API signature.

git diff HEAD~1 ./libicu/ndk_headers/

May need to run this to update .patch files

# Manually update doxygen doc in the .h headers in libicu/ndk_headers
./tools/icu4c_srcgen/doc_patches/create_patches.sh
git add -A && git commit  -F- <<EOF
Regenerate patches in libicu headers
EOF

5g. [Not required for every ICU release] Increment Distro major version

See https://android.googlesource.com/platform/system/timezone/+/main/README.android for details. Usually, it’s needed only when it’s the first ICU upgrade in the Android dessert release.

5h. Generate time zone files

cd $ANDROID_BUILD_TOP/system/timezone
repo start ${ICU_BRANCH} .
./update-tzdata.py && \
git add -A && \
git commit -F- <<EOF
Regenerate data files for ICU ${ICU_VERSION} upgrade

Binary data files updated using:
system/timezone/update-tzdata.py

This updates the ICU version number inside of icu_tzdata.dat but doesn't
change the timezone data itself.

Bug: ${ICU_UPGRADE_BUG}
Test: n/a
EOF

# If only the build time in icu4c/source/data/misc/zoneinfo64.txt, this step isn't needed. 
cd $ANDROID_BUILD_TOP/external/icu
git add -A
git commit -F- <<EOF
Regenerate tz-related data files

Data files updated using:
system/timezone/update-tzdata.py

Bug: ${ICU_UPGRADE_BUG}
Test: n/a
EOF

Ideally, it should not generate new time zone data files if tzdata has been updated for a new tzdb release
If it updates time zone data, there could be extra changes from CLDR. Talk to android-libcore-team@ how to deal with the updates, because they may need to cherry-pick them into tzdata module updates.

5i. Regenerate frameworks/base/libs/androidfw/LocaleDataLookup.cpp

croot frameworks/base
repo start icu${ICU_VERSION} .
./tools/localedata/extract_icu_data.py $ANDROID_BUILD_TOP > libs/androidfw/LocaleDataLookup.cpp
git commit -a -F- <<EOF
Regenerate LocaleDataLookup.cpp due to ICU ${ICU_VERSION} upgrade

The command:
./tools/localedata/extract_icu_data.py \$ANDROID_BUILD_TOP > libs/androidfw/LocaleDataLookup.cpp

Bug: ${ICU_UPGRADE_BUG}
Test: atest libandroidfw_tests FrameworksCoreTests:android.text.format
EOF

Build and run test
6a. Build by m droid cts
6b. Run the device tests by atest CtsIcu4cTestCases CtsIcuTestCases CtsLibcoreTestCases CtsLibcoreOjTestCases CtsBionicTestCases CtsTextTestCases minikin_tests -- --abi x86_64 # the primary ABI
- If running CtsIcu4cTestCases causes a native crash, e.g. exit code 137, check adb logcat for native crash dump.
- If the native crash dump isn't found, try to run the following shell command before checking the logcat again.
- ```
m CtsIcu4cTestCases && \
adb push out/soong/.intermediates/external/icu/icu4c/source/test/intltest/intltest64/android_x86_64_silvermont/intltest64 /data/local/tmp/intltest && \ # replace android_x86_64_silvermont if you are not using cuttlefish.
adb push out/host/linux-x86/testcases/CtsIcu4cTestCases/icu4c_test_data.zip /data/local/tmp/icu4c_test_data.zip && \
adb shell unzip -o -d /data/local/tmp/ /data/local/tmp/icu4c_test_data.zip && \
adb shell chmod a+x /data/local/tmp/intltest && \
adb shell /data/local/tmp/intltest -w -x /data/local/tmp/intltest_res.xml
```
6c. [No longer needed] Run the host-side test by
- ICU4J host-side test ant check
- ICU4C host-side test make CINTLTST_OPTS=-w INTLTEST_OPTS=-w check (Currently, it has some failing tests. No of failures?)
6d. If libcore/ tests are changed, verify the change
- To verify the test change in ART MTS, run m mts && mts-tradefed run mts-art on Android S.
- To verify the test change on LUCI bot, run art/tools/run-libcore-tests.sh --mode=host, because LUCI uses older ICU versions.
Upload the CLs to gerrit for code reviews from goog/icu-staging in external/icu and goog/upstream-release-cldr in external/cldr
```
repo upload --cbr -o nokeycheck -o uploadvalidator~skip --no-verify .
```

Merge goog/icu* branch to goog/main

cd $ANDROID_BUILD_TOP/external/icu
repo start icu${ICU_VERSION}-main .
git merge --no-ff icu-staging -m "
Merge branch goog/icu-staging into goog/main

Bug: ${ICU_UPGRADE_BUG}
Test: atest CtsIcu4cTestCases CtsIcuTestCases CtsLibcoreTestCases CtsLibcoreOjTestCases CtsBionicTestCases CtsTextTestCases minikin_tests
"

Update the version numbers in the METADATA 9a. Update the external/icu/METADATA in git_main 9b. Update version and upgrade date in external/cldr/METADATA 9c. git commit the file change
Upload and submit changes from external/icu, external/cldr, libcore, frameworks/base, system/timezone
10a. repo upload --cbr -o uploadvalidator~skip --no-verify .
10b. Code review the diff in android_icu4j/src/main/tests/android/icu/extratest/expected_transliteration_id_list.txt
- If a transliteration id is removed from the list, it may introduce app compatibility issues if the app depends on them. However, app has been warned to check the availability before invoking them. https://developer.android.com/reference/android/icu/text/Transliterator#getAvailableIDs()

After submitting all the CLs to goog/main, expose the new stable ICU4J APIs to Android SDK

rm tools/srcgen/allowlisted-public-api.txt
./tools/srcgen/generate_android_icu4j.sh
# Modify Icu4jTransform.java to allowlist more classes if needed. Check the error message for the details
m update-api droid

git commit -a -F- <<EOF
Expose the new stable APIs from ICU4J ${ICU_VERSION}

According to the upstream API coverage report external/icu/icu4j/coverage-exclusion.txt,
the methods should have API coverage running in the existing CtsIcuTestCases.

Bug: ${ICU_UPGRADE_BUG}
Test: atest CtsIcuTestCases
EOF

Send email android-libcore@ and icu-team@ to announce this.
1. Some Android teams may have dependency on the new ICU version for other features.
2. Email template

Hi, Libcore and ICU team,

ICU <version> just landed Android AOSP.
https://googleplex-android.googlesource.com/platform/external/icu/+/main/METADATA

Note:
- Contains bug fixes / build changes with a small set of API methods added.
- Unicode stays at version 16, and no new version has been published yet.

Historic version of this doc

http://g3doc/third_party/icu/g3doc/update/android