tree e3875107c5ddd686ab1a05fdd664fb11f172467c
parent f68fc8df0458a1c713d9b93f831a78871becb091
author Luis Hector Chavez <lhchavez@google.com> 1573815405 -0800
committer Treehugger Robot <treehugger-gerrit@google.com> 1574111023 +0000

Prevent identifiers with prefixes that are valid tokens from being split

This change ensures that identifiers that have a prefix that is an
otherwise valid token (such as `inotify_wait` that has a prefix `in`,
which is an OP) from tokenizing the prefix as a separate token.

Bug: chromium:1024021
Test: ./tools/compiler_unittest.py
Test: ./tools/parser_unittest.py

Change-Id: Ieae168c573116314bcdd7e862448c8ee298b05d4