doc/Test-Writing-Guidelines.asciidoc - platform/external/ltp - Git at Google

 LTP Test Writing Guidelines
 ===========================

 This document describes LTP guidelines and is intended for anybody who want to
 write or modify a LTP testcase. It's not a definitive guide and it's not, by
 any means, a substitute for common sense.

 NOTE: See also
       https://github.com/linux-test-project/ltp/wiki/C-Test-API[C Test API],
       https://github.com/linux-test-project/ltp/wiki/Shell-Test-API[Shell Test API],
       https://github.com/linux-test-project/ltp/wiki/LTP-Library-API-Writing-Guidelines[LTP Library API Writing Guidelines].

 Rules and recommendations which are "machine checkable" should be
 tagged with an ID like +LTP-XXX+. There will be a corresponding entry
 in
 https://github.com/linux-test-project/ltp/tree/master/doc/rules.tsv[doc/rules.tsv]. When
 you run 'make check' or 'make check-test' it will display these IDs as
 a reference.

 1. Guide to clean and understandable code
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 For testcases it's required that the source code is as easy to follow as
 possible. When a test starts to fail the failure has to be analyzed, clean
 test codebase makes this task much easier and quicker.

 Here are some hints on how to write clean and understandable code, a few of
 these points are further discussed below:

 * First of all *Keep things simple*

 * Keep function and variable names short but descriptive

 * Keep functions reasonably short and focused on a single task

 * Do not overcomment

 * Be consistent

 * Avoid deep nesting

 * DRY

 1.1 Keep things simple
 ~~~~~~~~~~~~~~~~~~~~~~

 For all it's worth keep the testcases simple or better as simple as possible.

 The kernel and libc are tricky beasts and the complexity imposed by their
 interfaces is quite high. Concentrate on the interface you want to test and
 follow the UNIX philosophy.

 It's a good idea to make the test as self-contained as possible too, ideally
 tests should not depend on tools or libraries that are not widely available.

 Do not reinvent the wheel!

 * Use LTP standard interface

 * Do not add custom PASS/FAIL reporting functions

 * Do not write Makefiles from scratch, use LTP build system instead

 * Etc.

 1.2 Keep functions and variable names short
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 Choosing a good name for an API functions or even variables is a difficult
 task do not underestimate it.

 There are a couple of customary names for different things that help people to
 understand code, for example:

 * For loop variables are usually named with a single letter 'i', 'j', ...

 * File descriptors 'fd' or 'fd_foo'.

 * Number of bytes stored in file are usually named as 'size' or 'len'

 * Etc.

 1.3 Do not overcomment
 ~~~~~~~~~~~~~~~~~~~~~~

 Comments can sometimes save you day but they can easily do more harm than
 good. There has been several cases where comments and actual implementation
 drifted slowly apart which yielded into API misuses and hard to find bugs.
 Remember there is only one thing worse than no documentation, wrong
 documentation.

 Ideally everybody should write code that is obvious, which unfortunately isn't
 always possible. If there is a code that requires to be commented keep it
 short and to the point. These comments should explain *why* and not *how*
 things are done.

 Never ever comment the obvious.

 In case of LTP testcases it's customary to add an asciidoc formatted comment
 paragraph with highlevel test description at the beginning of the file right
 under the GPL SPDX header. This helps other people to understand the overall
 goal of the test before they dive into the technical details. It's also
 exported into generated documentation hence it should mostly explain what is
 tested.

 1.4 DRY (Code duplication)
 ~~~~~~~~~~~~~~~~~~~~~~~~~~

 Copy & paste is a good servant but very poor master. If you are about to copy a
 large part of the code from one testcase to another, think what would happen if
 you find bug in the code that has been copied all around the tree. What about
 moving it to a library instead?

 The same goes for short but complicated parts, whenever you are about to copy &
 paste a syscall wrapper that packs arguments accordingly to machine
 architecture or similarly complicated code, put it into a header instead.

 2 Coding style
 ~~~~~~~~~~~~~~

 2.1 C coding style
 ^^^^^^^^^^^^^^^^^^

 LTP adopted Linux kernel coding style:
 https://www.kernel.org/doc/html/latest/process/coding-style.html

 If you aren't familiar with its rules please read it, it's a well written
 introduction.

 Run `make check` in the test's directory and/or use `make check-$TCID`,
 it uses (among other checks) our vendored version of
 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/scripts/checkpatch.pl[checkpatch.pl]
 script from kernel git tree.

 NOTE: If `make check` does not report any problems, the code still may be wrong
       as all tools used for checking only look for common mistakes.

 2.1.1 LTP-004: Test executable symbols are marked static
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

 Test executables should not export symbols unnecessarily. This means
 that all top-level variables and functions should be marked with the
 static keyword. The only visible symbols should be those included from
 shared object files.

 2.2 Shell coding style
 ^^^^^^^^^^^^^^^^^^^^^^

 When writing testcases in shell write in *portable shell* only, it's a good
 idea to try to run the test using alternative shell (alternative to bash, for
 example dash) too.

 *Portable shell* means Shell Command Language as defined by POSIX with an
 exception of few widely used extensions, namely 'local' keyword used inside of
 functions and '-o' and '-a' test parameters (that are marked as obsolete in
 POSIX).

 You can either try to run the testcases on Debian which has '/bin/sh' pointing
 to 'dash' by default or install 'dash' on your favorite distribution and use
 it to run the tests. If your distribution lacks 'dash' package you can always
 compile it from http://gondor.apana.org.au/~herbert/dash/files/[source].

 Run `make check` in the test's directory and/or use `make check-$TCID.sh`,
 it uses (among other checks) our vendored version of
 https://salsa.debian.org/debian/devscripts/raw/master/scripts/checkbashisms.pl[checkbashism.pl]
 from Debian, that is used to check for non-portable shell code.

 NOTE: If `make check` does not report any problems, the code still may be wrong
       as `checkbashisms.pl` used for checking only looks for common mistakes.

 Here are some common sense style rules for shell

 * Keep lines under 80 chars

 * Use tabs for indentation

 * Keep things simple, avoid unnecessary subshells

 * Don't do confusing things (i.e. don't name your functions like common shell
   commands, etc.)

 * Quote variables

 * Be consistent

 3 Backwards compatibility
 ~~~~~~~~~~~~~~~~~~~~~~~~~

 LTP test should be as backward compatible as possible. Think of an enterprise
 distributions with long term support (more than five years since the initial
 release) or of an embedded platform that needs to use several years old
 toolchain supplied by the manufacturer.

 Therefore LTP test for more current features should be able to cope with older
 systems. It should at least compile fine and if it's not appropriate for the
 configuration it should return 'TCONF'.

 There are several types of checks we use:

 The *configure script* is usually used to detect availability of a function
 declarations in system headers. It's used to disable tests at compile time or
 to enable fallback definitions.

 Checking the *errno* value is another type of runtime check. Most of the
 syscalls returns either 'EINVAL' or 'ENOSYS' when syscall was not implemented
 or was disabled upon kernel compilation.

 LTP has kernel version detection that can be used to disable tests at runtime,
 unfortunately kernel version does not always corresponds to a well defined
 feature set as distributions tend to backport hundreds of patches while the
 kernel version stays the same. Use with caution.

 Lately we added kernel '.config' parser, a test can define a boolean
 expression of kernel config variables that has to be satisfied in order for a
 test to run. This is mostly used for kernel namespaces at the moment.

 Sometimes it also makes sense to define a few macros instead of creating
 configure test. One example is Linux specific POSIX clock ids in
 'include/lapi/posix_clocks.h'.

 3.1 Dealing with messed up legacy code
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 LTP still contains a lot of old and messy code and we are cleaning it up as
 fast as we can but despite the decade of efforts there is still a lot. If you
 start modifying old or a messy testcase and your changes are more complicated
 than simple typo fixes you should convert the test into a new library first.

 It's also much easier to review the changes if you split them into a smaller
 logical groups. The same goes for moving files. If you need a rename or move
 file do it in a separate patch.

 4 License
 ~~~~~~~~~

 Code contributed to LTP should be licensed under GPLv2+ (GNU GPL version 2 or
 any later version).

 Use `SPDX-License-Identifier: GPL-2.0-or-later`

 5 LTP Structure
 ~~~~~~~~~~~~~~~

 The structure of LTP is quite simple. Each test is a binary written either in
 portable shell or C. The test gets a configuration via environment variables
 and/or command line parameters, it prints additional information into the
 stdout and reports overall success/failure via the exit value.

 Tests are generally placed under the 'testcases/' directory. Everything that
 is a syscall or (slightly confusingly) libc syscall wrapper goes under
 'testcases/kernel/syscalls/'.

 Then there is 'testcases/open_posix_testsuite/' which is a well maintained fork
 of the upstream project that has been dead since 2005 and also a number of
 directories with tests for more specific features.

 5.1 Runtest Files
 ^^^^^^^^^^^^^^^^^

 The list of tests to be executed is stored in runtest files under the
 'runtest/' directory. The default set of runtest files to be executed is
 stored in 'scenario_groups/default'. When you add a test you should add
 corresponding entries into some runtest file(s) as well.

 Each line of runtest file contain one test. First item is the test name
 ('shell_test01', 'splice02'). All other items, separated by space will be
 executed ('echo "SUCCESS" | shell_pipe01.sh', 'splice02 -s 20').

 [source,sh]
 -------------------------------------------------------------------------------
 shell_test01 echo "SUCCESS" | shell_pipe01.sh
 splice02 splice02 -s 20
 -------------------------------------------------------------------------------

 Blank lines and lines starting with a '#' (comments) are ignored.

 For syscall tests (these placed under 'testcases/kernel/syscalls/') use
 'runtest/syscalls' file, for kernel related tests for memory management we
 have 'runtest/mm', etc.

 IMPORTANT: The runtest files should have one entry per a test. Creating a
            wrapper that runs all your tests and adding it as a single test
            into runtest file is strongly discouraged.

 5.2 Datafiles
 ^^^^^^^^^^^^^

 If your test needs datafiles to work, these should be put into a subdirectory
 named 'datafiles' and installed into the 'testcases/data/$TCID' directory (to
 do that you have to add 'INSTALL_DIR := testcases/data/TCID' into the
 'datafiles/Makefile').

 You can obtain path to datafiles via $TST_DATAROOT provided by test.sh
 '$TST_DATAROOT/...'
 or via C function 'tst_dataroot()' provided by libltp:

 [source,c]
 -------------------------------------------------------------------------------
 const char *dataroot = tst_dataroot();
 -------------------------------------------------------------------------------

 Datafiles can also be accessed as '$LTPROOT/testcases/data/$TCID/...',
 but '$TST_DATAROOT' and 'tst_dataroot()' are preferred as these can be used
 when running testcases directly in git tree as well as from install
 location.

 The path is constructed according to these rules:

 1. if '$LTPROOT' is set, return '$LTPROOT/testcases/data/$TCID'
 2. else if 'tst_tmpdir()' was called return '$STARTWD/datafiles'
    (where '$STARTWD' is initial working directory as recorded by 'tst_tmpdir()')
 3. else return '$CWD/datafiles'

 See 'testcases/commands/file/' for example.

 5.3 Subexecutables
 ^^^^^^^^^^^^^^^^^^

 If your test needs to execute a binary, place it in the same directory as the
 testcase and name the file starting with '${test_binary_name}_'.  Once the
 test is executed by the framework, the path to the directory with all LTP
 binaries is added to the '$PATH' and you can execute it just by its name.

 TIP: If you need to execute such test from the LTP tree, you can add path to
      current directory to '$PATH' manually with: 'PATH="$PATH:$PWD" ./foo01'.

 6 Test Contribution Checklist
 ------------------------------

 NOTE: See also
       https://github.com/linux-test-project/ltp/wiki/Maintainer-Patch-Review-Checklist[Maintainer Patch Review Checklist].

 1. Test compiles and runs fine (check with `-i 10` too)
 2. `make check` does not emit any warnings for the test you are working on
    (hint: run it in the test's directory and/or use `make check-$TCID`)
 3. The runtest entries are in place
 4. Test binaries are added into corresponding '.gitignore' files
 5. Patches apply over the latest git

 6.1 About .gitignore files
 ~~~~~~~~~~~~~~~~~~~~~~~~~~

 There are numerous '.gitignore' files in the LTP tree. Usually there is a
 '.gitignore' file per a group of tests. The reason for this setup is simple.
 It's easier to maintain a '.gitignore' file per directory with tests, rather
 than having single file in the project root directory. This way, we don't have
 to update all the gitignore files when moving directories, and they get deleted
 automatically when a directory with tests is removed.

 7 Testing pre-release kernel features
 -------------------------------------

 Tests for features not yet in a mainline kernel release are accepted. However
 they must only be added to the +staging+ runtest file. Once a feature is part
 of the stable kernel ABI the associated test must be moved out of staging.

 This is primarily to help test kernel RCs by avoiding the need to download
 separate LTP patchsets.

 8 LTP C And Shell Test API Comparison
 -------------------------------------

 Comparison of
 https://github.com/linux-test-project/ltp/wiki/C-Test-API[C Test API] and
 https://github.com/linux-test-project/ltp/wiki/Shell-Test-API[Shell Test API].

 [options="header"]
 |================================================================================
 |  C API ('struct tst_test' members) | shell API ('$TST_*' environment variables)
 | '.all_filesystems' | 'TST_ALL_FILESYSTEMS'
 | '.bufs' | –
 | '.caps' | –
 | '.child_needs_reinit' | not applicable
 | '.cleanup' | 'TST_CLEANUP'
 | '.dev_extra_opts' | 'TST_DEV_EXTRA_OPTS'
 | '.dev_fs_opts' | 'TST_DEV_FS_OPTS'
 | '.dev_fs_type' | 'TST_FS_TYPE'
 | '.dev_min_size' | not applicable
 | '.format_device' | 'TST_FORMAT_DEVICE'
 | '.max_runtime' | –
 | '.min_cpus' | not applicable
 | '.min_kver' | 'TST_MIN_KVER'
 | '.min_mem_avail' | not applicable
 | '.mnt_flags' | 'TST_MNT_PARAMS'
 | '.min_swap_avail' | not applicable
 | '.mntpoint', '.mnt_data' | 'TST_MNTPOINT'
 | '.mount_device' | 'TST_MOUNT_DEVICE'
 | '.needs_cgroup_ctrls' | –
 | '.needs_checkpoints' | 'TST_NEEDS_CHECKPOINTS'
 | '.needs_cmds' | 'TST_NEEDS_CMDS'
 | '.needs_devfs' | –
 | '.needs_device' | 'TST_NEEDS_DEVICE'
 | '.needs_drivers' | 'TST_NEEDS_DRIVERS'
 | '.needs_kconfigs' | 'TST_NEEDS_KCONFIGS'
 | '.needs_overlay' |
 | '.needs_rofs' | –
 | '.needs_root' | 'TST_NEEDS_ROOT'
 | '.needs_tmpdir' | 'TST_NEEDS_TMPDIR'
 | '.options' | 'TST_PARSE_ARGS', 'TST_OPTS'
 | '.resource_files' | –
 | '.restore_wallclock' | not applicable
 | '.sample' | –
 | '.save_restore' | –
 | '.scall' | not applicable
 | '.setup' | 'TST_SETUP'
 | '.skip_filesystems' | 'TST_SKIP_FILESYSTEMS'
 | '.skip_in_compat' | –
 | '.skip_in_lockdown' | 'TST_SKIP_IN_LOCKDOWN'
 | '.skip_in_secureboot' | 'TST_SKIP_IN_SECUREBOOT'
 | '.supported_archs' | not applicable
 | '.tags' | –
 | '.taint_check' | –
 | '.tcnt' | 'TST_CNT'
 | '.tconf_msg' | not applicable
 | '.test', '.test_all' | 'TST_TESTFUNC'
 | '.test_variants' | –
 | '.timeout' | 'TST_TIMEOUT'
 | '.tst_hugepage' | not applicable
 | .format_device | 'TST_DEVICE'
 | not applicable | 'TST_NEEDS_KCONFIGS_IFS'
 | not applicable | 'TST_NEEDS_MODULE'
 | not applicable | 'TST_POS_ARGS'
 | not applicable | 'TST_USAGE'
 |================================================================================
	LTP Test Writing Guidelines
	===========================

	This document describes LTP guidelines and is intended for anybody who want to
	write or modify a LTP testcase. It's not a definitive guide and it's not, by
	any means, a substitute for common sense.

	NOTE: See also
	https://github.com/linux-test-project/ltp/wiki/C-Test-API[C Test API],
	https://github.com/linux-test-project/ltp/wiki/Shell-Test-API[Shell Test API],
	https://github.com/linux-test-project/ltp/wiki/LTP-Library-API-Writing-Guidelines[LTP Library API Writing Guidelines].

	Rules and recommendations which are "machine checkable" should be
	tagged with an ID like +LTP-XXX+. There will be a corresponding entry
	in
	https://github.com/linux-test-project/ltp/tree/master/doc/rules.tsv[doc/rules.tsv]. When
	you run 'make check' or 'make check-test' it will display these IDs as
	a reference.

	1. Guide to clean and understandable code
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

	For testcases it's required that the source code is as easy to follow as
	possible. When a test starts to fail the failure has to be analyzed, clean
	test codebase makes this task much easier and quicker.

	Here are some hints on how to write clean and understandable code, a few of
	these points are further discussed below:

	* First of all Keep things simple

	* Keep function and variable names short but descriptive

	* Keep functions reasonably short and focused on a single task

	* Do not overcomment

	* Be consistent

	* Avoid deep nesting

	* DRY

	1.1 Keep things simple
	~~~~~~~~~~~~~~~~~~~~~~

	For all it's worth keep the testcases simple or better as simple as possible.

	The kernel and libc are tricky beasts and the complexity imposed by their
	interfaces is quite high. Concentrate on the interface you want to test and
	follow the UNIX philosophy.

	It's a good idea to make the test as self-contained as possible too, ideally
	tests should not depend on tools or libraries that are not widely available.

	Do not reinvent the wheel!

	* Use LTP standard interface

	* Do not add custom PASS/FAIL reporting functions

	* Do not write Makefiles from scratch, use LTP build system instead

	* Etc.

	1.2 Keep functions and variable names short
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

	Choosing a good name for an API functions or even variables is a difficult
	task do not underestimate it.

	There are a couple of customary names for different things that help people to
	understand code, for example:

	* For loop variables are usually named with a single letter 'i', 'j', ...

	* File descriptors 'fd' or 'fd_foo'.

	* Number of bytes stored in file are usually named as 'size' or 'len'

	* Etc.

	1.3 Do not overcomment
	~~~~~~~~~~~~~~~~~~~~~~

	Comments can sometimes save you day but they can easily do more harm than
	good. There has been several cases where comments and actual implementation
	drifted slowly apart which yielded into API misuses and hard to find bugs.
	Remember there is only one thing worse than no documentation, wrong
	documentation.

	Ideally everybody should write code that is obvious, which unfortunately isn't
	always possible. If there is a code that requires to be commented keep it
	short and to the point. These comments should explain why and not how
	things are done.

	Never ever comment the obvious.

	In case of LTP testcases it's customary to add an asciidoc formatted comment
	paragraph with highlevel test description at the beginning of the file right
	under the GPL SPDX header. This helps other people to understand the overall
	goal of the test before they dive into the technical details. It's also
	exported into generated documentation hence it should mostly explain what is
	tested.

	1.4 DRY (Code duplication)
	~~~~~~~~~~~~~~~~~~~~~~~~~~

	Copy & paste is a good servant but very poor master. If you are about to copy a
	large part of the code from one testcase to another, think what would happen if
	you find bug in the code that has been copied all around the tree. What about
	moving it to a library instead?

	The same goes for short but complicated parts, whenever you are about to copy &
	paste a syscall wrapper that packs arguments accordingly to machine
	architecture or similarly complicated code, put it into a header instead.

	2 Coding style
	~~~~~~~~~~~~~~

	2.1 C coding style
	^^^^^^^^^^^^^^^^^^

	LTP adopted Linux kernel coding style:
	https://www.kernel.org/doc/html/latest/process/coding-style.html

	If you aren't familiar with its rules please read it, it's a well written
	introduction.

	Run `make check` in the test's directory and/or use `make check-$TCID`,
	it uses (among other checks) our vendored version of
	https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/scripts/checkpatch.pl[checkpatch.pl]
	script from kernel git tree.

	NOTE: If `make check` does not report any problems, the code still may be wrong
	as all tools used for checking only look for common mistakes.

	2.1.1 LTP-004: Test executable symbols are marked static
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

	Test executables should not export symbols unnecessarily. This means
	that all top-level variables and functions should be marked with the
	static keyword. The only visible symbols should be those included from
	shared object files.

	2.2 Shell coding style
	^^^^^^^^^^^^^^^^^^^^^^

	When writing testcases in shell write in portable shell only, it's a good
	idea to try to run the test using alternative shell (alternative to bash, for
	example dash) too.

	Portable shell means Shell Command Language as defined by POSIX with an
	exception of few widely used extensions, namely 'local' keyword used inside of
	functions and '-o' and '-a' test parameters (that are marked as obsolete in
	POSIX).

	You can either try to run the testcases on Debian which has '/bin/sh' pointing
	to 'dash' by default or install 'dash' on your favorite distribution and use
	it to run the tests. If your distribution lacks 'dash' package you can always
	compile it from http://gondor.apana.org.au/~herbert/dash/files/[source].

	Run `make check` in the test's directory and/or use `make check-$TCID.sh`,
	it uses (among other checks) our vendored version of
	https://salsa.debian.org/debian/devscripts/raw/master/scripts/checkbashisms.pl[checkbashism.pl]
	from Debian, that is used to check for non-portable shell code.

	NOTE: If `make check` does not report any problems, the code still may be wrong
	as `checkbashisms.pl` used for checking only looks for common mistakes.

	Here are some common sense style rules for shell

	* Keep lines under 80 chars

	* Use tabs for indentation

	* Keep things simple, avoid unnecessary subshells

	* Don't do confusing things (i.e. don't name your functions like common shell
	commands, etc.)

	* Quote variables

	* Be consistent

	3 Backwards compatibility
	~~~~~~~~~~~~~~~~~~~~~~~~~

	LTP test should be as backward compatible as possible. Think of an enterprise
	distributions with long term support (more than five years since the initial
	release) or of an embedded platform that needs to use several years old
	toolchain supplied by the manufacturer.

	Therefore LTP test for more current features should be able to cope with older
	systems. It should at least compile fine and if it's not appropriate for the
	configuration it should return 'TCONF'.

	There are several types of checks we use:

	The configure script is usually used to detect availability of a function
	declarations in system headers. It's used to disable tests at compile time or
	to enable fallback definitions.

	Checking the errno value is another type of runtime check. Most of the
	syscalls returns either 'EINVAL' or 'ENOSYS' when syscall was not implemented
	or was disabled upon kernel compilation.

	LTP has kernel version detection that can be used to disable tests at runtime,
	unfortunately kernel version does not always corresponds to a well defined
	feature set as distributions tend to backport hundreds of patches while the
	kernel version stays the same. Use with caution.

	Lately we added kernel '.config' parser, a test can define a boolean
	expression of kernel config variables that has to be satisfied in order for a
	test to run. This is mostly used for kernel namespaces at the moment.

	Sometimes it also makes sense to define a few macros instead of creating
	configure test. One example is Linux specific POSIX clock ids in
	'include/lapi/posix_clocks.h'.

	3.1 Dealing with messed up legacy code
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

	LTP still contains a lot of old and messy code and we are cleaning it up as
	fast as we can but despite the decade of efforts there is still a lot. If you
	start modifying old or a messy testcase and your changes are more complicated
	than simple typo fixes you should convert the test into a new library first.

	It's also much easier to review the changes if you split them into a smaller
	logical groups. The same goes for moving files. If you need a rename or move
	file do it in a separate patch.

	4 License
	~~~~~~~~~

	Code contributed to LTP should be licensed under GPLv2+ (GNU GPL version 2 or
	any later version).

	Use `SPDX-License-Identifier: GPL-2.0-or-later`

	5 LTP Structure
	~~~~~~~~~~~~~~~

	The structure of LTP is quite simple. Each test is a binary written either in
	portable shell or C. The test gets a configuration via environment variables
	and/or command line parameters, it prints additional information into the
	stdout and reports overall success/failure via the exit value.

	Tests are generally placed under the 'testcases/' directory. Everything that
	is a syscall or (slightly confusingly) libc syscall wrapper goes under
	'testcases/kernel/syscalls/'.

	Then there is 'testcases/open_posix_testsuite/' which is a well maintained fork
	of the upstream project that has been dead since 2005 and also a number of
	directories with tests for more specific features.

	5.1 Runtest Files
	^^^^^^^^^^^^^^^^^

	The list of tests to be executed is stored in runtest files under the
	'runtest/' directory. The default set of runtest files to be executed is
	stored in 'scenario_groups/default'. When you add a test you should add
	corresponding entries into some runtest file(s) as well.

	Each line of runtest file contain one test. First item is the test name
	('shell_test01', 'splice02'). All other items, separated by space will be
	executed ('echo "SUCCESS" \| shell_pipe01.sh', 'splice02 -s 20').

	[source,sh]
	-------------------------------------------------------------------------------
	shell_test01 echo "SUCCESS" \| shell_pipe01.sh
	splice02 splice02 -s 20
	-------------------------------------------------------------------------------

	Blank lines and lines starting with a '#' (comments) are ignored.

	For syscall tests (these placed under 'testcases/kernel/syscalls/') use
	'runtest/syscalls' file, for kernel related tests for memory management we
	have 'runtest/mm', etc.

	IMPORTANT: The runtest files should have one entry per a test. Creating a
	wrapper that runs all your tests and adding it as a single test
	into runtest file is strongly discouraged.

	5.2 Datafiles
	^^^^^^^^^^^^^

	If your test needs datafiles to work, these should be put into a subdirectory
	named 'datafiles' and installed into the 'testcases/data/$TCID' directory (to
	do that you have to add 'INSTALL_DIR := testcases/data/TCID' into the
	'datafiles/Makefile').

	You can obtain path to datafiles via $TST_DATAROOT provided by test.sh
	'$TST_DATAROOT/...'
	or via C function 'tst_dataroot()' provided by libltp:

	[source,c]
	-------------------------------------------------------------------------------
	const char *dataroot = tst_dataroot();
	-------------------------------------------------------------------------------

	Datafiles can also be accessed as '$LTPROOT/testcases/data/$TCID/...',
	but '$TST_DATAROOT' and 'tst_dataroot()' are preferred as these can be used
	when running testcases directly in git tree as well as from install
	location.

	The path is constructed according to these rules:

	1. if '$LTPROOT' is set, return '$LTPROOT/testcases/data/$TCID'
	2. else if 'tst_tmpdir()' was called return '$STARTWD/datafiles'
	(where '$STARTWD' is initial working directory as recorded by 'tst_tmpdir()')
	3. else return '$CWD/datafiles'

	See 'testcases/commands/file/' for example.

	5.3 Subexecutables
	^^^^^^^^^^^^^^^^^^

	If your test needs to execute a binary, place it in the same directory as the
	testcase and name the file starting with '${test_binary_name}_'. Once the
	test is executed by the framework, the path to the directory with all LTP
	binaries is added to the '$PATH' and you can execute it just by its name.

	TIP: If you need to execute such test from the LTP tree, you can add path to
	current directory to '$PATH' manually with: 'PATH="$PATH:$PWD" ./foo01'.

	6 Test Contribution Checklist
	------------------------------

	NOTE: See also
	https://github.com/linux-test-project/ltp/wiki/Maintainer-Patch-Review-Checklist[Maintainer Patch Review Checklist].

	1. Test compiles and runs fine (check with `-i 10` too)
	2. `make check` does not emit any warnings for the test you are working on
	(hint: run it in the test's directory and/or use `make check-$TCID`)
	3. The runtest entries are in place
	4. Test binaries are added into corresponding '.gitignore' files
	5. Patches apply over the latest git

	6.1 About .gitignore files
	~~~~~~~~~~~~~~~~~~~~~~~~~~

	There are numerous '.gitignore' files in the LTP tree. Usually there is a
	'.gitignore' file per a group of tests. The reason for this setup is simple.
	It's easier to maintain a '.gitignore' file per directory with tests, rather
	than having single file in the project root directory. This way, we don't have
	to update all the gitignore files when moving directories, and they get deleted
	automatically when a directory with tests is removed.

	7 Testing pre-release kernel features
	-------------------------------------

	Tests for features not yet in a mainline kernel release are accepted. However
	they must only be added to the +staging+ runtest file. Once a feature is part
	of the stable kernel ABI the associated test must be moved out of staging.

	This is primarily to help test kernel RCs by avoiding the need to download
	separate LTP patchsets.

	8 LTP C And Shell Test API Comparison
	-------------------------------------

	Comparison of
	https://github.com/linux-test-project/ltp/wiki/C-Test-API[C Test API] and
	https://github.com/linux-test-project/ltp/wiki/Shell-Test-API[Shell Test API].

	[options="header"]
	\|================================================================================
	\| C API ('struct tst_test' members) \| shell API ('$TST_*' environment variables)
	\| '.all_filesystems' \| 'TST_ALL_FILESYSTEMS'
	\| '.bufs' \| –
	\| '.caps' \| –
	\| '.child_needs_reinit' \| not applicable
	\| '.cleanup' \| 'TST_CLEANUP'
	\| '.dev_extra_opts' \| 'TST_DEV_EXTRA_OPTS'
	\| '.dev_fs_opts' \| 'TST_DEV_FS_OPTS'
	\| '.dev_fs_type' \| 'TST_FS_TYPE'
	\| '.dev_min_size' \| not applicable
	\| '.format_device' \| 'TST_FORMAT_DEVICE'
	\| '.max_runtime' \| –
	\| '.min_cpus' \| not applicable
	\| '.min_kver' \| 'TST_MIN_KVER'
	\| '.min_mem_avail' \| not applicable
	\| '.mnt_flags' \| 'TST_MNT_PARAMS'
	\| '.min_swap_avail' \| not applicable
	\| '.mntpoint', '.mnt_data' \| 'TST_MNTPOINT'
	\| '.mount_device' \| 'TST_MOUNT_DEVICE'
	\| '.needs_cgroup_ctrls' \| –
	\| '.needs_checkpoints' \| 'TST_NEEDS_CHECKPOINTS'
	\| '.needs_cmds' \| 'TST_NEEDS_CMDS'
	\| '.needs_devfs' \| –
	\| '.needs_device' \| 'TST_NEEDS_DEVICE'
	\| '.needs_drivers' \| 'TST_NEEDS_DRIVERS'
	\| '.needs_kconfigs' \| 'TST_NEEDS_KCONFIGS'
	\| '.needs_overlay' \|
	\| '.needs_rofs' \| –
	\| '.needs_root' \| 'TST_NEEDS_ROOT'
	\| '.needs_tmpdir' \| 'TST_NEEDS_TMPDIR'
	\| '.options' \| 'TST_PARSE_ARGS', 'TST_OPTS'
	\| '.resource_files' \| –
	\| '.restore_wallclock' \| not applicable
	\| '.sample' \| –
	\| '.save_restore' \| –
	\| '.scall' \| not applicable
	\| '.setup' \| 'TST_SETUP'
	\| '.skip_filesystems' \| 'TST_SKIP_FILESYSTEMS'
	\| '.skip_in_compat' \| –
	\| '.skip_in_lockdown' \| 'TST_SKIP_IN_LOCKDOWN'
	\| '.skip_in_secureboot' \| 'TST_SKIP_IN_SECUREBOOT'
	\| '.supported_archs' \| not applicable
	\| '.tags' \| –
	\| '.taint_check' \| –
	\| '.tcnt' \| 'TST_CNT'
	\| '.tconf_msg' \| not applicable
	\| '.test', '.test_all' \| 'TST_TESTFUNC'
	\| '.test_variants' \| –
	\| '.timeout' \| 'TST_TIMEOUT'
	\| '.tst_hugepage' \| not applicable
	\| .format_device \| 'TST_DEVICE'
	\| not applicable \| 'TST_NEEDS_KCONFIGS_IFS'
	\| not applicable \| 'TST_NEEDS_MODULE'
	\| not applicable \| 'TST_POS_ARGS'
	\| not applicable \| 'TST_USAGE'
	\|================================================================================