tree 8c25cb51f25029565c38864f19761909d70afd70
parent 8ac906faf7b66180f2208380c35ae1e07136c5cc
author jshin@chromium.org <jshin@chromium.org> 1413241608 +0000
committer jshin@chromium.org <jshin@chromium.org> 1413241608 +0000

Make all the single byte encodings compliant to the encoding spec.

1. Replace the current encoding alias list (heavily patched) with our own
HTML5-specific alias list. It's mostly generated from encoding.json, which
is in turn derived from the WHATWG Encoding living standard. The most notable
difference is that UTF-32 entries are kept until bug 417850 is resolved.
Two other differences are:

 a. Two aliases for iso-8859-8-i (logical and csiso88598i) are not listed.
 They're dealt with in Blink.

 b. Chinese (gb*, big5*) aliases are not yet aligned to the encoding
spec pending our decision on the unification of Big5 / Big5-HKSCS and
GBK / GB18030.

2. Replace all the single-byte mapping tables with what's automatically
generated with scripts/single-byte-gen.sh that uses index-* files downloaded
from the WHATWG spec site. This will fix the decoding (ToUnicode) of
windows-874 and windows-1253 while removing a lot of fallback/spurrious
mapping entries in encoding direction ('FromUnicode') in a number of encodings.

3. Regenerate the ICU binary data files for Linux/Mac/Android/Windows/CrOS.

4. Remove now obsolete noop-*ucm files used to make ISO-2022-CN* decoder
to turn an empty string. They're not necessary any more because ISO-2022-CN*
were made 'replacement' encodings in Blink and our version of ICU does not
have any code for ISO-2022-CN* any more.

This cuts down the data size by 15kB. On Android, there's virtually
no change in the data size because the previous data file on Android
accidentally had smaller locale data for nb and ms.

BUG=412053
TEST=browser_tests --gtest_filter="*ncoding*"
TEST=net_unittest --gtest_filter="*ilenameUtil*"
TEST=base_unittests --gtest_filter="*Conv*"
TEST=Blink: fast/encoding/*
TEST=http://www.w3.org/International/tests/repository/encoding/indexes/results-indexes
TEST=http://www.w3.org/International/tests/repository/encoding/indexes/results-aliases
TEST=http://www.w3.org/International/tests/repository/run?manifest=encoding/indexes&test=windows-1253_test
TEST=http://www.w3.org/International/tests/repository/run?manifest=encoding/indexes&test=windows-874_test
R=jsbell@chromium.org

Review URL: https://codereview.chromium.org/598383002

git-svn-id: http://src.chromium.org/svn/trunk/deps/third_party/icu52@292447 4ff67af0-8c30-449e-8e8b-ad334ec8d88c