Google Git
Sign in
android/platform/external/python/cpython3/ab0ad620383/./Tools/cases_generator
tree: caec4355212ee39cbb0030ac284cc064123cd14f [path history] [tgz]
  1. _typing_backports.py
  2. analyzer.py
  3. cwriter.py
  4. generators_common.py
  5. interpreter_definition.md
  6. lexer.py
  7. mypy.ini
  8. opcode_id_generator.py
  9. opcode_metadata_generator.py
  10. parser.py
  11. parsing.py
  12. plexer.py
  13. py_metadata_generator.py
  14. README.md
  15. stack.py
  16. target_generator.py
  17. tier1_generator.py
  18. tier2_generator.py
  19. uop_id_generator.py
  20. uop_metadata_generator.py
Tools/cases_generator/README.md

Tooling to generate interpreters

Documentation for the instruction definitions in Python/bytecodes.c (“the DSL”) is here.

What's currently here:

  • lexer.py: lexer for C, originally written by Mark Shannon
  • plexer.py: OO interface on top of lexer.py; main class: PLexer
  • parsing.py: Parser for instruction definition DSL; main class Parser
  • generate_cases.py: driver script to read Python/bytecodes.c and write Python/generated_cases.c.h (and several other files)
  • analysis.py: Analyzer class used to read the input files
  • flags.py: abstractions related to metadata flags for instructions
  • formatting.py: Formatter class used to write the output files
  • instructions.py: classes to analyze and write instructions
  • stacking.py: code to handle generalized stack effects

Note that there is some dummy C code at the top and bottom of Python/bytecodes.c to fool text editors like VS Code into believing this is valid C code.

A bit about the parser

The parser class uses a pretty standard recursive descent scheme, but with unlimited backtracking. The PLexer class tokenizes the entire input before parsing starts. We do not run the C preprocessor. Each parsing method returns either an AST node (a Node instance) or None, or raises SyntaxError (showing the error in the C source).

Most parsing methods are decorated with @contextual, which automatically resets the tokenizer input position when None is returned. Parsing methods may also raise SyntaxError, which is irrecoverable. When a parsing method returns None, it is possible that after backtracking a different parsing method returns a valid AST.

Neither the lexer nor the parsers are complete or fully correct. Most known issues are tersely indicated by # TODO: comments. We plan to fix issues as they become relevant.