commit	7012bf923fe5493014479a0627f11d7316ac2008	[log] [tgz]
author	David Neto <dneto@google.com>	Wed Jun 14 10:38:32 2017 -0400
committer	David Neto <dneto@google.com>	Wed Jun 14 10:39:10 2017 -0400
tree	ded2607d471675085ef333320d66833135443982
parent	ed3b9e19db04d6ef7a495370211576426c350530 [diff]
parent	91c46c656720a6e1e71a3411cd1f4f792b427b2d [diff]

Merge remote-tracking branch 'aosp/upstream-master' into update-shaderc

Includes:
91c46c65 Merge pull request #932 from LoopDawg/warning-fix-1
1892886a HLSL: compilation warning fix: no functional change
5ea6a192 Merge branch 'amdrexu-feature'
cabbb788 Implement extension GL_AMD_gpu_shader_int16
4d5bcd31 HLSL: Allow macro expansions to create the 'defined' operator.
3fcb42cf GLSL: Fix #853: Only outer dimension of array can be specialization constant.
d314ecfb GLSL: Correct missing "not" from error message about non writeonly images.
dd56173d GLSL: Disallow 'shared' in nested scopes.
c5f2ab96 GLSL: Disallow unsized arrays of atomic_uint.
885443c7 Build: reset for multi-threaded test error, that reported a single missing character.
14d65beb GLSL: Fix Khronos private Bugzilla 15973: local sizes can't be 0.
2eb13550 GLSL: Fix #396: Error when 'defined' comes from macro expansion.
65755667 Merge pull request #922 from KhronosGroup/dash-I
971a0a8d Infrastructure: add include search paths (-I, etc.).
96f65521 HLSL: Implement half matrices, and map all half* -> float*.
f6deacd5 HLSL: Track control-flow nesting and warn on aliasing under it.
9b2531ba Infrastructure: Move nesting counters, etc., to base class.
b4d46627 Build: Fix build warnings.
a83959fa Merge pull request #920 from jeremy-lunarg/jeremy-build
549764b5 Build: Fix linux
0e07119a HLSL: Fix #919: for-init-statement is arbitrary declaration or expression.
e00e8f45 GLSL: replace general missing functionality with specific messages.
21369c82 Merge pull request #912 from amdrexu/feature
225e0fca Implement the extension GL_AMD_texture_gather_bias_lod
94c18a84 GLSL: Fix bug setting component=0 for an auto-location assignment.
8de7e7bf GLSL: Error when using location on an arrayed block.
99f289d4 Merge pull request #899 from antiagainst/rm-empty-cpps
0b94a31e Merge pull request #915 from LoopDawg/subvec4-intrinsic
70942e5f Merge pull request #905 from KhronosGroup/flatten-opaque-structs
f3150742 HLSL: Convert run-time sampler assignments to compile-time aliases.
750c2d07 SPV: When passing structs of opaque types, flatten and pass the members instead.
d66c5b12 HLSL: iomapper: Fix #914. Tolerate user aliasing of bindings.
a696fd1a HLSL: add test coverage for sub-vec4 texture intrinsics
ae99875e Merge pull request #907 from KhronosGroup/include
5052152e Merge pull request #913 from amdrexu/bugfix
3494b4da HLSL: Add an Includer to handle #include.
0848cc07 Parser: Add missing codes for float16
44d2728e Merge pull request #911 from KhronosGroup/debug-info
e485c7af SPV: Debug output: Include OpLine information for execution path.
121853f4 SPV: Add OpSource shader source code and file name.
136b1e2d Merge pull request #856 from TiemoJung/texture_upgrade
baf570ef Pure Texture to Sampled Texture Transform
d6af18f6 Merge pull request #901 from LoopDawg/imat-construct
7c9129bc SPV: Fix #904: Correctly check for built-in block redeclations for location check.
84cc15f0 HLSL: Fix #903: Don't short-circuit && or ||.
6e2295d3 HLSL: Fix #902: Incorrect protection against zero arguments.
174ccb8f HLSL: Add imat, umat, and bmat constructors
2c6f48b1 Fix #373: Implicitly make gl_FragColor a location=0 output.
1d585ac8 SPV: Correctly enforce 'location' presence on in/out blocks.
557caf24 Merge pull request #900 from LoopDawg/tx-overloads
7b8c386c More non-determinism fixed.
b6cabc4f Fix C++ portability, non-deterministic order of argument evaluation.
80f92a19 Fix #857: Convert uniform int to local bool for struct alias assignment.
0d2b4713 HLSL: Don't do logical short-circuits when the operands are bool-vectors.
132a28aa HLSL: allow name mangling based on texture template type
ddda9cb8 Remove empty cpp files
ab0847ef Merge pull request #896 from KhronosGroup/spv-location
ff164d87 Merge pull request #897 from LoopDawg/remap-specconstop-fix.2
65c2eed6 Remapper: handle embedded opcode in OpSpecConstantOp
71facdf4 SPV: Give error on not assigning locations to I/O.
24e895b4 Merge pull request #860 from steve-lunarg/sb-counter-args.2
09a29d9b Merge pull request #893 from KhronosGroup/member-call
2bb1f39f WIP: HLSL: add ability to pass struct buffers with counters to fns
0a2a0cd3 HLSL: Implement member functions calling member functions.
b29cc30c Merge pull request #852 from steve-lunarg/declared-builtin
a4bfed12 WIP: track declared builtin type [proposal]
1a010b83 Merge pull request #890 from LoopDawg/mip-operator
726bf96a HLSL: add .mips[][] operator for texture types
c70bbc83 Merge pull request #889 from antiagainst/travis-android
b21213e8 Check Android build on Travis CI.
ef7efb21 Merge pull request #888 from LoopDawg/geometry-stage-mix
c6510a33 HLSL: allow GS-specific methods in other stages
c49b4efa HLSL: Fix #884: Use promoted children, not pre-promoted, in completing binary nodes.
c48c8e76 Merge pull request #886 from antiagainst/unordered-set
2840f63f Use unordered_set in ReadableOrderTraverser.
d9b08d5c Merge pull request #880 from aras-p/master
02c4728a Merge pull request #874 from xlpiao/DescriptorSet-and-Binding
23d3c712 [lumped builds] Add include guards (#pragma once) to header files that did not have any.
8e204b2d [lumped builds] Only define _CRT_SECURE_NO_WARNINGS if it’s not defined yet. When glslang is built with some other build system and lumped/unity builds are used, without the checks this would get “macro is being redefined” warnings/errors.
36dc8290 HLSL: Manually configure descriptor set and binding number for resources
12d69368 Build fix: Make string assignment simpler to avoid some compiler issues.
1ee1c8e1 Build: Fix portability in previous commit.
c10191d1 SPV: Support texelFetch() on a textureBuffer (no sampler).
a5c5fb61 SPV: Emit extension for using StorageClassStorageBuffer.
43c72b0e Merge pull request #878 from steve-lunarg/unroll-hint
f1709e71 HLSL: implement [unroll] and [loop] attributes
de1cc06c Merge pull request #877 from steve-lunarg/warn-fix1
0c20067c Fix iomapper build warnings
7aaf4191 Merge pull request #873 from steve-lunarg/gathercmp
6817f81e HLSL: Implement missing GatherCmp
22be5788 Merge pull request #870 from SoapGentoo/cmake-fixes
22afc38b Modernise CMake #2
eb9a532b Merge pull request #869 from KhronosGroup/revert-828-fixes
cfc69d95 Revert "Modernise CMake"
a01600b8 Merge pull request #828 from SoapGentoo/fixes
8f824265 Make test suite optional by using CTest
5a5699bd Modernise CMake
acce8dce Merge pull request #868 from TiemoJung/two_pass_io_remap
bbae7de0 Merge pull request #867 from steve-lunarg/emptyinit
f1bfeec7 Notification phase for io remapper
c0043cda HLSL: Allow empty struct initializers
7cca1409 HLSL: Turn on hlsl-offsets by default for HLSL-source shaders.
dd5dee0b Merge pull request #859 from steve-lunarg/sampleposition
d4d0b297 HLSL: add standard sample position return form of GetSamplePosition method
88aabcca Merge pull request #855 from steve-lunarg/constantbuffer
a766b838 HLSL: Add ConstantBuffer<T> syntax
a1cdd13b Merge pull request #851 from dgkoch/dgkoch_build_fixes3
15bb4370 Fix Android build errors
8a5718a9 Merge pull request #849 from steve-lunarg/samplercmp-errmsg
3cbc32f4 HLSL: add error for expected comparison sampler in SampleCmp* ops
32a385e9 HLSL: Fix #846: support mixed ternary types.
0603a383 Merge pull request #847 from steve-lunarg/sb-param-fix
1c04f1e5 Merge pull request #842 from steve-lunarg/sb-cast
1487db54 Merge pull request #840 from steve-lunarg/iomap-hlsl
e404e088 HLSL: fix for byte address buffers in fn parmeters
7b1dcd66 HLSL: add readonly qualifier to tbuffer, so they end up as SRV
be283550 WIP: HLSL: hlsl register class iomapping
ba5cc2fa GLSL: Fix #822: Improve information given for syntax errors.
0c6f9360 HLSL: Fix #96: Support do-while loop substatements with no curly braces.
f8203a0a HLSL: cast non-int types to uint on Load/Store indexes
67027189 SPV: Support test #pragma for generating the StorageBuffer storage class.
a8d3db6b Merge pull request #835 from steve-lunarg/sb-counters
d5d9ffbd HLSL: vector shape conversions for all ops: Fix #839. Fix #653. Fix #631.
2aa12b1c HLSL: Address #839: avoid crash by distinguishing between bad argument and no argument.
350b9485 WIP: HLSL: add refection queries for structuredbuffer counter blocks
12bc9aa9 WIP: HLSL: add Append/ConsumeBuffer support
8e26feb8 WIP: HLSL: structuredbuffer counter functionality
d1141843 SPV Non-functional: fix ordering causing problems with another branch.
97366a0d HLSL: Fix #770: implicitly convert bool operands to numeric operators.
a4c64c98 HLSL: Fix #834: Report #version is an illegal command.
a0c578a6 HLSL: Fix #758: Support character literals (except for numeric escape sequences).
2051815b HLSL: Fix #803: Add shape conversions to the constant-initializer path.
a8b217fd Merge pull request #833 from KhronosGroup/hlsl-inf-syntax
776c515e HLSL: Support 1.#INF and -1.#INF syntax.
b5e739c2 HLSL: Fix boolean conversion bug and add more tests for ?:.
636b62db HLSL: Support vector 'cond ? :' -> EOpMix -> OpSelect.
34718204 HLSL: Non-functional: add vector ? : tests.
13075c61 HLSL: Fix #832: don't require terminating semicolon for cbuffer/tbuffer.
e9f18fd9 Merge pull request #831 from steve-lunarg/isfinite
9e5a19fd HLSL: fix return type for isfinite
ae79697d Merge pull request #827 from amdrexu/feature4
a09eefd8 Non-functional: Fix typos in comments.
6fa17641 HLSL: Emit the OpSource HLSL instruction for HLSL, using new headers.
7962bda3 Merge pull request #825 from steve-lunarg/index-cast
f89ad980 Implement the extension SPV_KHR_16bit_storage
2efd6c6d HLSL: cast bracket dereference index to int type if not.
38a42535 SPV: Move to latest public headers.
07b30c42 Merge pull request #823 from steve-lunarg/tess-coord-size
4f1403ed SPV: Fix #807: use --hlsl-offsets to allow hlsl-style offsets in a buffer.
ccb076ac HLSL: allow non-vec3 tessellation coordinate declarations
6f1e595d Merge branch 'jantlo-cpp11-feat'
d92b5679 Merge branch 'cpp11-feat' of https://github.com/jantlo/glslang into jantlo-cpp11-feat
a558b265 This refactor the StandAlone and WorkList files in order to use more C++11 features remove the dependencies from OS specific code. Changes:
97cb85c7 PP: Fix #783: Catch end-of-argument expansion in token pasting.
82460b5e HLSL: Fix #805: Support cast of scalars to structures.
5ce1e4af Merge pull request #817 from steve-lunarg/isfinite
c633f644 HLSL: Non-functional: rationalize making constructors.
8df9a486 Merge pull request #812 from steve-lunarg/PatchIO
13975525 Decompose OpIsFinite to avoid capability restrictions
067eb9b4 WIP: HLSL: Support InputPatch variables in patch constant functions
b68b9a8b Merge pull request #808 from steve-lunarg/gs-ep-wrap-fix
08e0c086 HLSL: fix GS implementation for EP wrapping
2184c2f2 Fix #287: pin down the io-array size before checking for out of range index.
6f03bfc7 Merge pull request #815 from steve-lunarg/tess-ds-pcf-order
f38cca3c HLSL: handle PCF input to DS in arbitrary argument position
fa84001a Merge pull request #814 from steve-lunarg/contains
27309f68 Refactor TType::contains* methods (nonfunctional)
c7fd73b7 Merge pull request #811 from steve-lunarg/combine-test
bf1537f4 WIP: HLSL: force uncombined flag off for Buffer<>
f36542f4 Revert "Merge pull request #779 from steve-lunarg/buffer-unsampled-fix"
84a30c8b Merge pull request #774 from steve-lunarg/tess-ctrlpt-pcf
e70fcf2d Merge pull request #810 from steve-lunarg/clip-crash-fix
db2e3b41 HLSL: fix crash on empty sequence node passed to intrinsic expansions
e741249b HLSL: pass tessellation execution modes through to SPIR-V
9ce76553 Merge pull request #797 from steve-lunarg/scalar-length
18958f6c HLSL: Fix #802: Preserve promoted child under ! operator.
7e997e26 HLSL: Implicit bool conversions for conditional expressions and related.
8f9fdc98 HLSL: Add namespace grammar and some basic semantics.
e752f463 HLSL: HS return is arrayed to match SPIR-V semantics
7afe1344 HLSL: strip off array dimension when assign locations of arrayed IO.
194f0f39 HLSL: require tessellation factors to be fixed size arrays
9cee73e0 HLSL: support per control point patch const fn invocation
e434ad92 Fix #809: smear scalar condition in OpSelect for selecting vector operands.
4dc835c3 Non-functional: Round of adding 'const', related to more efficient getFullNamespaceName().
1ca04c2b HLSL: allow length() on scalars

Test: checkbuild.py on Linux; unit tests on Windows
Change-Id: Ia1cca8c3f03f62303899e4df0cff13b47d9d6ea5

tree: ded2607d471675085ef333320d66833135443982

README.md

Also see the Khronos landing page for glslang as a reference front end:

https://www.khronos.org/opengles/sdk/tools/Reference-Compiler/

The above page includes where to get binaries, and is kept up to date regarding the feature level of glslang.

glslang

An OpenGL and OpenGL ES shader front end and validator.

There are several components:

A GLSL/ESSL front-end for reference validation and translation of GLSL/ESSL into an AST.
An HLSL front-end for translation of a broad generic HLL into the AST. See issue 362 and issue 701 for current status.
A SPIR-V back end for translating the AST to SPIR-V.
A standalone wrapper, glslangValidator, that can be used as a command-line tool for the above.

How to add a feature protected by a version/extension/stage/profile: See the comment in glslang/MachineIndependent/Versions.cpp.

Tasks waiting to be done are documented as GitHub issues.

Execution of Standalone Wrapper

To use the standalone binary form, execute glslangValidator, and it will print a usage statement. Basic operation is to give it a file containing a shader, and it will print out warnings/errors and optionally an AST.

The applied stage-specific rules are based on the file extension:

.vert for a vertex shader
.tesc for a tessellation control shader
.tese for a tessellation evaluation shader
.geom for a geometry shader
.frag for a fragment shader
.comp for a compute shader

There is also a non-shader extension

.conf for a configuration file of limits, see usage statement for example

Building

Dependencies

CMake: for generating compilation targets.
bison: optional, but needed when changing the grammar (glslang.y).
googletest: optional, but should use if making any changes to glslang.

Build steps

1) Check-Out this project

cd <parent of where you want glslang to be>
# If using SSH
git clone git@github.com:KhronosGroup/glslang.git
# Or if using HTTPS
git clone https://github.com/KhronosGroup/glslang.git

2) Check-Out External Projects

cd <the directory glslang was cloned to, "External" will be a subdirectory>
git clone https://github.com/google/googletest.git External/googletest

3) Configure

Assume the source directory is $SOURCE_DIR and the build directory is $BUILD_DIR:

For building on Linux (assuming using the Ninja generator):

cd $BUILD_DIR

cmake -GNinja -DCMAKE_BUILD_TYPE={Debug|Release|RelWithDebInfo} \
      -DCMAKE_INSTALL_PREFIX=`pwd`/install $SOURCE_DIR

For building on Windows:

cmake $SOURCE_DIR -DCMAKE_INSTALL_PREFIX=`pwd`/install
# The CMAKE_INSTALL_PREFIX part is for testing (explained later).

The CMake GUI also works for Windows (version 3.4.1 tested).

4) Build and Install

# for Linux:
ninja install

# for Windows:
cmake --build . --config {Release|Debug|MinSizeRel|RelWithDebInfo} \
      --target install

If using MSVC, after running CMake to configure, use the Configuration Manager to check the INSTALL project.

If you need to change the GLSL grammar

The grammar in glslang/MachineIndependent/glslang.y has to be recompiled with bison if it changes, the output files are committed to the repo to avoid every developer needing to have bison configured to compile the project when grammar changes are quite infrequent. For windows you can get binaries from GnuWin32.

The command to rebuild is:

bison --defines=MachineIndependent/glslang_tab.cpp.h
      -t MachineIndependent/glslang.y
      -o MachineIndependent/glslang_tab.cpp

The above command is also available in the bash script at glslang/updateGrammar.

Testing

Right now, there are two test harnesses existing in glslang: one is Google Test, one is the runtests script. The former runs unit tests and single-shader single-threaded integration tests, while the latter runs multiple-shader linking tests and multi-threaded tests.

Running tests

The runtests script requires compiled binaries to be installed into $BUILD_DIR/install. Please make sure you have supplied the correct configuration to CMake (using -DCMAKE_INSTALL_PREFIX) when building; otherwise, you may want to modify the path in the runtests script.

Running Google Test-backed tests:

cd $BUILD_DIR

# for Linux:
ctest

# for Windows:
ctest -C {Debug|Release|RelWithDebInfo|MinSizeRel}

# or, run the test binary directly
# (which gives more fine-grained control like filtering):
<dir-to-glslangtests-in-build-dir>/glslangtests

Running runtests script-backed tests:

cd $SOURCE_DIR/Test && ./runtests

Contributing tests

Test results should always be included with a pull request that modifies functionality.

If you are writing unit tests, please use the Google Test framework and place the tests under the gtests/ directory.

Integration tests are placed in the Test/ directory. It contains test input and a subdirectory baseResults/ that contains the expected results of the tests. Both the tests and baseResults/ are under source-code control.

Google Test runs those integration tests by reading the test input, compiling them, and then compare against the expected results in baseResults/. The integration tests to run via Google Test is registered in various gtests/*.FromFile.cpp source files. glslangtests provides a command-line option --update-mode, which, if supplied, will overwrite the golden files under the baseResults/ directory with real output from that invocation. For more information, please check gtests/ directory's README.

For the runtests script, it will generate current results in the localResults/ directory and diff them against the baseResults/. When you want to update the tracked test results, they need to be copied from localResults/ to baseResults/. This can be done by the bump shell script.

You can add your own private list of tests, not tracked publicly, by using localtestlist to list non-tracked tests. This is automatically read by runtests and included in the diff and bump process.

Programmatic Interfaces

Another piece of software can programmatically translate shaders to an AST using one of two different interfaces:

A new C++ class-oriented interface, or
The original C functional interface

The main() in StandAlone/StandAlone.cpp shows examples using both styles.

C++ Class Interface (new, preferred)

This interface is in roughly the last 1/3 of ShaderLang.h. It is in the glslang namespace and contains the following.

const char* GetEsslVersionString();
const char* GetGlslVersionString();
bool InitializeProcess();
void FinalizeProcess();

class TShader
    bool parse(...);
    void setStrings(...);
    const char* getInfoLog();

class TProgram
    void addShader(...);
    bool link(...);
    const char* getInfoLog();
    Reflection queries

See ShaderLang.h and the usage of it in StandAlone/StandAlone.cpp for more details.

C Functional Interface (orignal)

This interface is in roughly the first 2/3 of ShaderLang.h, and referred to as the Sh*() interface, as all the entry points start Sh.

The Sh*() interface takes a “compiler” call-back object, which it calls after building call back that is passed the AST and can then execute a backend on it.

The following is a simplified resulting run-time call stack:

ShCompile(shader, compiler) -> compiler(AST) -> <back end>

In practice, ShCompile() takes shader strings, default version, and warning/error and other options for controlling compilation.

Basic Internal Operation

Initial lexical analysis is done by the preprocessor in MachineIndependent/Preprocessor, and then refined by a GLSL scanner in MachineIndependent/Scan.cpp. There is currently no use of flex.
Code is parsed using bison on MachineIndependent/glslang.y with the aid of a symbol table and an AST. The symbol table is not passed on to the back-end; the intermediate representation stands on its own. The tree is built by the grammar productions, many of which are offloaded into ParseHelper.cpp, and by Intermediate.cpp.
The intermediate representation is very high-level, and represented as an in-memory tree. This serves to lose no information from the original program, and to have efficient transfer of the result from parsing to the back-end. In the AST, constants are propogated and folded, and a very small amount of dead code is eliminated.
To aid linking and reflection, the last top-level branch in the AST lists all global symbols.
The primary algorithm of the back-end compiler is to traverse the tree (high-level intermediate representation), and create an internal object code representation. There is an example of how to do this in MachineIndependent/intermOut.cpp.
Reduction of the tree to a linear byte-code style low-level intermediate representation is likely a good way to generate fully optimized code.
There is currently some dead old-style linker-type code still lying around.
Memory pool: parsing uses types derived from C++ std types, using a custom allocator that puts them in a memory pool. This makes allocation of individual container/contents just few cycles and deallocation free. This pool is popped after the AST is made and processed.
The use is simple: if you are going to call new, there are three cases:
- the object comes from the pool (its base class has the macro POOL_ALLOCATOR_NEW_DELETE in it) and you do not have to call delete
- it is a TString, in which case call NewPoolTString(), which gets it from the pool, and there is no corresponding delete
- the object does not come from the pool, and you have to do normal C++ memory management of what you new