commit	a14954cb8fd4b7b6030dfae3a77ef926e12f3a7a	[log] [tgz]
author	David Neto <dneto@google.com>	Tue Apr 17 18:11:59 2018 -0400
committer	David Neto <dneto@google.com>	Tue Apr 17 18:12:38 2018 -0400
tree	52515d5aae4f529cc992ca7110b4c6d86c7f6c0e
parent	a413185bd9b036524aceae81c39f752c0edd7536 [diff]
parent	3bb4c48cd95892a5cfcd63df20fcc47fd51c97a0 [diff]

Merge remote-tracking branch 'aosp/upstream-master' into update-shaderc

Includes:
3bb4c48c Bump revision.
16b80f78 Merge pull request #1345 from amdrexu/bugfix
7c88aff1 Specify SPV_AMD_gpu_shader_int16 extension string when frexp() takes 16-bit exp operand
6cf7645c Bump revision.
c034c5a8 HLSL/SPV: Declare DepthReplacing for shaders declaring SV_Depth* outputs.
24eae066 Merge pull request #1348 from greg-lunarg/kg16
d91e0360 Merge pull request #1351 from danginsburg/issue_1350
5a69b675 Update gold results
95c87b65 Fix issue #1350 - set the node type to match the atomic TIntermAggregate node type so that AtomicSMin/Max is not generated with a result type of void.
0f8d43e5 GLSL/Vulkan: Enforce no location aliasing in vertex inputs.
c51bff8d Update spirv-opt known good
5d3babdb Build: Fix #1347: Compile with NV_EXTENSIONS disabled.
b9e9e57f Merge pull request #1326 from mattparks/patch1
461ea099 Non-functional: Fix tabs.
e891afac GLSL: Fix #1330: default outputs for GL_NV_geometry_shader_passthrough
9de57c81 Bump revision.
5cd155f0 Merge pull request #1343 from KhronosGroup/uint-counter-bufs
c3a370db Merge pull request #1341 from St0fF-NPL-ToM/master
6ae18707 HLSL: Fix #1332: consistently use uint for counter buf typing.
1aaa3567 Revert "add ability to record accessed and declared "named defines""
619dc896 Merge pull request #1338 from cgmb/minor-shaderlang-fixups
1dcd1623 GLSL/SPV: no location on atomic_uint; Fix #1339.
39bbad5a Minor cleanup in ShaderLang.cpp
a1184ddd fix further MSVC 2012 build issues
e7350423 add ability to record accessed and declared "named defines"
77d04598 fix VS2012 compile issues
b92ce60f GLSL/SPV: Fix #1310: don't create unnecessary integer matrices.
88a6a181 README: Expand on the C++11 compiler requirement for MSVS.
11b5c311 Update revision.
5d8d7882 Fix #1328: ENABLE_HLSL=OFF breaking build.
f52b6381 Fix #1331: Emit SPV_GOOGLE_hlsl_functionality1 for counters.
7f0bcfd8 Fix #1333: Protect against -g for non-generating code.
a89f8cf7 Merge branch 'nonuniform-dynindex'
4597fea8 Update revision.
6a8a678c Revision/Version update.
5611c6d2 GLSL/SPV: Implement SPV_EXT_descriptor_indexing and GL_EXT_nonuniform_qualifier
0b5e5da7 Update revision.
6a4a427e GLSL: Implement correct semantic checking for run-time sized arrays.
5a867aca Linker: Fix #1329: correctly merge unsized arrays, and fix link tests.
ed834895 Merge pull request #1327 from mattparks/patch2
21993569 Added getter for uniform block bindings
6c5f649a Allowed for LIB_TYPE to be set to switch between static and shared libs for glslang, SPIRV, and HLSL
fbdba369 Merge pull request #1325 from KhronosGroup/simplify-doc
10971553 Non-Functional: Remove unused parts of the disassembler.
d688796a Merge pull request #1322 from jeffbolznv/master
2abe9a4d Implement GL_NV_shader_subgroup_partitioned
6b1ede41 Merge pull request #1320 from greg-lunarg/addopt21
fb03a55e Only print legalization warning if optimizer disabled
7be36647 Merge pull request #1319 from KhronosGroup/array-rationalization
f9a05f96 Merge pull request #1316 from greg-lunarg/addopt18
c408b430 Merge pull request #1318 from amdrexu/feature
ead86224 Non-functional. Rationalizations enabling future generalizations:
2316924b Non-Functional: Remove merge-replicated code and fix white space.
ae06d1fe Add int16 support to AMD_trinary_minmax and AMD_shader_ballot
4bede9eb Disable DCE by Remapper through glslangValidator...
643e57cd Merge pull request #1313 from greg-lunarg/kg15
26ad99b5 Merge pull request #1312 from KhronosGroup/fix-mixed-array-styles
0b46c1ed Merge branch 'master' into kg15
eecb874c Update SPIRV-Tools known good.
859b0342 Types: Fix #1290: Rationalize and correct "mixed" style array dimensioning.
1c3ab274 Test Results Only: Include size in implicitly sized arrays (change to unsized).
7cec64fc Non functional: add EshTargetClientVersion alias for EShTargetClientVersion.
97e35420 Non-functional: always return value from makeFpConstant().
569dd84a Merge pull request #1306 from Igalia/uniform-aml-v2
1b1ad97d Merge pull request #1305 from amdrexu/bugfix
2d539049 Take into account the number of locations taken up by a uniform
1fe4a447 Add locations to opaque types for OpenGL
b126550b Add additional error check for fragment shader outputs
95b1334a Bump patch version.
ca4366cf Merge pull request #1302 from KhronosGroup/cpp-relax-skiptokens-errors
e7e081bd PP: don't give errors on some tokens under #if 0 (or similar).
56e80565 Merge pull request #1282 from KhronosGroup/hlslFunctionality1
5d610ee1 Implement SPV_GOOGLE_hlsl_functionality1.
cd23a475 Merge pull request #1291 from ben-clayton/add-nv-nopersp
0af69da1 Bump patch version.
fb1e40b4 Merge pull request #1299 from KhronosGroup/fix-conversions
f6480833 Add tests for GL_NV_shader_noperspective_interpolation
cb32c544 Merge pull request #1289 from Igalia/igalia/dmat-scalar-division
647fccaf AST: Fix shift conversions, which don't require matching types.
6e899be5 Non-functional: Rationalize operators handled by the split addConversion() functions.
137a24df AST: Remove convertToBasicType() which was obfuscating the conversion design.
075c8c11 Merge pull request #1294 from dgkoch/dgkoch_build_fixes4
bfe09521 Fix build several build errors
3e2a3c46 Merge pull request #1293 from baldurk/remove-en-dash
34cd7630 Replace en-dash with simple ASCII hyphen
017a567b Add support for GL_NV_shader_noperspective_interpolation
eddb1318 Use the correct type for the constant for matrix/scalar division
2ad4737d Merge pull request #1286 from greg-lunarg/kg14
b095da74 Merge pull request #1288 from amdrexu/bugfix
2bd8cd04 Merge pull request #1287 from eintw1ck/master
a00e51b5 HLSL: Correct some mistakes for min16 types
b3e80f5a include/Common.h: reorder includes to avoid redefinition on some platforms
bfa64afb Increment minor version to 4
c96885f4 Update known_good to SPIR-V 1.3 support

Change-Id: I33146ba342024fbef869ea20874713e7d1cec685
Testing: checkbuild.py on Linux; unit tests on Windows

tree: 52515d5aae4f529cc992ca7110b4c6d86c7f6c0e

README.md

Also see the Khronos landing page for glslang as a reference front end:

https://www.khronos.org/opengles/sdk/tools/Reference-Compiler/

The above page includes where to get binaries, and is kept up to date regarding the feature level of glslang.

glslang

An OpenGL and OpenGL ES shader front end and validator.

There are several components:

A GLSL/ESSL front-end for reference validation and translation of GLSL/ESSL into an AST.
An HLSL front-end for translation of a broad generic HLL into the AST. See issue 362 and issue 701 for current status.
A SPIR-V back end for translating the AST to SPIR-V.
A standalone wrapper, glslangValidator, that can be used as a command-line tool for the above.

How to add a feature protected by a version/extension/stage/profile: See the comment in glslang/MachineIndependent/Versions.cpp.

Tasks waiting to be done are documented as GitHub issues.

Execution of Standalone Wrapper

To use the standalone binary form, execute glslangValidator, and it will print a usage statement. Basic operation is to give it a file containing a shader, and it will print out warnings/errors and optionally an AST.

The applied stage-specific rules are based on the file extension:

.vert for a vertex shader
.tesc for a tessellation control shader
.tese for a tessellation evaluation shader
.geom for a geometry shader
.frag for a fragment shader
.comp for a compute shader

There is also a non-shader extension

.conf for a configuration file of limits, see usage statement for example

Building

Instead of building manually, you can also download the binaries for your platform directly from the master-tot release on GitHub. Those binaries are automatically uploaded by the buildbots after successful testing and they always reflect the current top of the tree of the master branch.

Dependencies

A C++11 compiler. (For MSVS: 2015 is recommended, 2013 is fully supported/tested, and 2010 support is attempted, but not tested.)
CMake: for generating compilation targets.
make: Linux, ninja is an alternative, if configured.
Python 2.7: for executing SPIRV-Tools scripts. (Optional if not using SPIRV-Tools.)
bison: optional, but needed when changing the grammar (glslang.y).
googletest: optional, but should use if making any changes to glslang.

Build steps

The following steps assume a Bash shell. On Windows, that could be the Git Bash shell or some other shell of your choosing.

1) Check-Out this project

cd <parent of where you want glslang to be>
git clone https://github.com/KhronosGroup/glslang.git

2) Check-Out External Projects

cd <the directory glslang was cloned to, "External" will be a subdirectory>
git clone https://github.com/google/googletest.git External/googletest

If you wish to assure that SPIR-V generated from HLSL is legal for Vulkan, or wish to invoke -Os to reduce SPIR-V size from HLSL or GLSL, install spirv-tools with this:

./update_glslang_sources.py

3) Configure

Assume the source directory is $SOURCE_DIR and the build directory is $BUILD_DIR. First ensure the build directory exists, then navigate to it:

mkdir -p $BUILD_DIR
cd $BUILD_DIR

For building on Linux:

cmake -DCMAKE_BUILD_TYPE={Debug|Release|RelWithDebInfo} \
      -DCMAKE_INSTALL_PREFIX="$(pwd)/install" $SOURCE_DIR

For building on Windows:

cmake $SOURCE_DIR -DCMAKE_INSTALL_PREFIX="$(pwd)/install"
# The CMAKE_INSTALL_PREFIX part is for testing (explained later).

The CMake GUI also works for Windows (version 3.4.1 tested).

4) Build and Install

# for Linux:
make -j4 install

# for Windows:
cmake --build . --config {Release|Debug|MinSizeRel|RelWithDebInfo} \
      --target install

If using MSVC, after running CMake to configure, use the Configuration Manager to check the INSTALL project.

If you need to change the GLSL grammar

The grammar in glslang/MachineIndependent/glslang.y has to be recompiled with bison if it changes, the output files are committed to the repo to avoid every developer needing to have bison configured to compile the project when grammar changes are quite infrequent. For windows you can get binaries from GnuWin32.

The command to rebuild is:

bison --defines=MachineIndependent/glslang_tab.cpp.h
      -t MachineIndependent/glslang.y
      -o MachineIndependent/glslang_tab.cpp

The above command is also available in the bash script at glslang/updateGrammar.

Testing

Right now, there are two test harnesses existing in glslang: one is Google Test, one is the runtests script. The former runs unit tests and single-shader single-threaded integration tests, while the latter runs multiple-shader linking tests and multi-threaded tests.

Running tests

The runtests script requires compiled binaries to be installed into $BUILD_DIR/install. Please make sure you have supplied the correct configuration to CMake (using -DCMAKE_INSTALL_PREFIX) when building; otherwise, you may want to modify the path in the runtests script.

Running Google Test-backed tests:

cd $BUILD_DIR

# for Linux:
ctest

# for Windows:
ctest -C {Debug|Release|RelWithDebInfo|MinSizeRel}

# or, run the test binary directly
# (which gives more fine-grained control like filtering):
<dir-to-glslangtests-in-build-dir>/glslangtests

Running runtests script-backed tests:

cd $SOURCE_DIR/Test && ./runtests

Contributing tests

Test results should always be included with a pull request that modifies functionality.

If you are writing unit tests, please use the Google Test framework and place the tests under the gtests/ directory.

Integration tests are placed in the Test/ directory. It contains test input and a subdirectory baseResults/ that contains the expected results of the tests. Both the tests and baseResults/ are under source-code control.

Google Test runs those integration tests by reading the test input, compiling them, and then compare against the expected results in baseResults/. The integration tests to run via Google Test is registered in various gtests/*.FromFile.cpp source files. glslangtests provides a command-line option --update-mode, which, if supplied, will overwrite the golden files under the baseResults/ directory with real output from that invocation. For more information, please check gtests/ directory's README.

For the runtests script, it will generate current results in the localResults/ directory and diff them against the baseResults/. When you want to update the tracked test results, they need to be copied from localResults/ to baseResults/. This can be done by the bump shell script.

You can add your own private list of tests, not tracked publicly, by using localtestlist to list non-tracked tests. This is automatically read by runtests and included in the diff and bump process.

Programmatic Interfaces

Another piece of software can programmatically translate shaders to an AST using one of two different interfaces:

A new C++ class-oriented interface, or
The original C functional interface

The main() in StandAlone/StandAlone.cpp shows examples using both styles.

C++ Class Interface (new, preferred)

This interface is in roughly the last 1/3 of ShaderLang.h. It is in the glslang namespace and contains the following.

const char* GetEsslVersionString();
const char* GetGlslVersionString();
bool InitializeProcess();
void FinalizeProcess();

class TShader
    setStrings(...);
    setEnvInput(EShSourceHlsl or EShSourceGlsl, stage,  EShClientVulkan or EShClientOpenGL, 100);
    setEnvClient(EShClientVulkan or EShClientOpenGL, EShTargetVulkan_1_0 or EShTargetVulkan_1_1 or EShTargetOpenGL_450);
    setEnvTarget(EShTargetSpv, EShTargetSpv_1_0 or EShTargetSpv_1_3);
    bool parse(...);
    const char* getInfoLog();

class TProgram
    void addShader(...);
    bool link(...);
    const char* getInfoLog();
    Reflection queries

See ShaderLang.h and the usage of it in StandAlone/StandAlone.cpp for more details.

C Functional Interface (orignal)

This interface is in roughly the first 2/3 of ShaderLang.h, and referred to as the Sh*() interface, as all the entry points start Sh.

The Sh*() interface takes a “compiler” call-back object, which it calls after building call back that is passed the AST and can then execute a backend on it.

The following is a simplified resulting run-time call stack:

ShCompile(shader, compiler) -> compiler(AST) -> <back end>

In practice, ShCompile() takes shader strings, default version, and warning/error and other options for controlling compilation.

Basic Internal Operation

Initial lexical analysis is done by the preprocessor in MachineIndependent/Preprocessor, and then refined by a GLSL scanner in MachineIndependent/Scan.cpp. There is currently no use of flex.
Code is parsed using bison on MachineIndependent/glslang.y with the aid of a symbol table and an AST. The symbol table is not passed on to the back-end; the intermediate representation stands on its own. The tree is built by the grammar productions, many of which are offloaded into ParseHelper.cpp, and by Intermediate.cpp.
The intermediate representation is very high-level, and represented as an in-memory tree. This serves to lose no information from the original program, and to have efficient transfer of the result from parsing to the back-end. In the AST, constants are propogated and folded, and a very small amount of dead code is eliminated.
To aid linking and reflection, the last top-level branch in the AST lists all global symbols.
The primary algorithm of the back-end compiler is to traverse the tree (high-level intermediate representation), and create an internal object code representation. There is an example of how to do this in MachineIndependent/intermOut.cpp.
Reduction of the tree to a linear byte-code style low-level intermediate representation is likely a good way to generate fully optimized code.
There is currently some dead old-style linker-type code still lying around.
Memory pool: parsing uses types derived from C++ std types, using a custom allocator that puts them in a memory pool. This makes allocation of individual container/contents just few cycles and deallocation free. This pool is popped after the AST is made and processed.
The use is simple: if you are going to call new, there are three cases:
- the object comes from the pool (its base class has the macro POOL_ALLOCATOR_NEW_DELETE in it) and you do not have to call delete
- it is a TString, in which case call NewPoolTString(), which gets it from the pool, and there is no corresponding delete
- the object does not come from the pool, and you have to do normal C++ memory management of what you new