tests/README: convert to markdown
Closes #6028
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 9773b86..7424fe6 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -25,7 +25,7 @@
MANDISTPAGES = runtests.1.dist testcurl.1.dist
EXTRA_DIST = ftpserver.pl httpserver.pl secureserver.pl runtests.pl \
- getpart.pm FILEFORMAT.md README stunnel.pem memanalyze.pl testcurl.pl \
+ getpart.pm FILEFORMAT.md README.md stunnel.pem memanalyze.pl testcurl.pl \
valgrind.pm ftp.pm sshserver.pl sshhelp.pm pathhelp.pm testcurl.1 runtests.1 \
serverhelp.pm tftpserver.pl rtspserver.pl directories.pm symbol-scan.pl \
CMakeLists.txt mem-include-scan.pl valgrind.supp extern-scan.pl \
diff --git a/tests/README b/tests/README
deleted file mode 100644
index 209887d..0000000
--- a/tests/README
+++ /dev/null
@@ -1,277 +0,0 @@
- _ _ ____ _
- ___| | | | _ \| |
- / __| | | | |_) | |
- | (__| |_| | _ <| |___
- \___|\___/|_| \_\_____|
-
-The curl Test Suite
-
- 1. Running
- 1.1 Requires to run
- 1.2 Port numbers used by test servers
- 1.3 Test servers
- 1.4 Run
- 1.5 Shell startup scripts
- 1.6 Memory test
- 1.7 Debug
- 1.8 Logs
- 1.9 Test input files
- 1.10 Code coverage
- 1.11 Remote testing
-
- 2. Numbering
- 2.1 Test case numbering
-
- 3. Write tests
- 3.1 test data
- 3.2 curl tests
- 3.3 libcurl tests
- 3.4 unit tests
-
- 4. TODO
- 4.1 More protocols
- 4.2 SOCKS auth
-
-==============================================================================
-
-1. Running
-
- 1.1 Requires to run
-
- perl (and a unix-style shell)
- python (and a unix-style shell, for SMB and TELNET tests)
- python-impacket (for SMB tests)
- diff (when a test fails, a diff is shown)
- stunnel (for HTTPS and FTPS tests)
- OpenSSH or SunSSH (for SCP, SFTP and SOCKS4/5 tests)
- nghttpx (for HTTP/2 tests)
- nroff (for --manual tests)
-
- 1.1.1 Installation of python-impacket
-
- The Python-based test servers support both recent Python 2 and 3.
- You can figure out your default Python interpreter with python -V
-
- Please install python-impacket in the correct Python environment.
- You can use pip or your OS' package manager to install 'impacket'.
-
- On Debian/Ubuntu the package names are:
- Python 2: 'python-impacket'
- Python 3: 'python3-impacket'
-
- On FreeBSD the package names are:
- Python 2: 'py27-impacket'
- Python 3: 'py37-impacket'
-
- On any system where pip is available:
- Python 2: 'pip2 install impacket'
- Python 3: 'pip3 install impacket'
-
- You may also need to manually install the Python package 'six'
- as that may be a missing requirement for impacket on Python 3.
-
- 1.2 Port numbers used by test servers
-
- Tests are written to use as few fixed fixed port numbers as possible and all
- tests should be written to use suitable variables instead of port numbers so
- that test cases continue to work independent on what port numbers the test
- servers actually use.
-
- The remaining fixed-port test servers that are still used, use the port
- range 8890 - 8904 by default, but can be moved with runtests' -b option.
-
- See the FILEFORMAT for a listing of existing port number variables.
-
- 1.3 Test servers
-
- The test suite runs simple FTP, POP3, IMAP, SMTP, HTTP and TFTP stand-alone
- servers on the ports listed above to which it makes requests. For SSL tests,
- it runs stunnel to handle encryption to the regular servers. For SSH, it
- runs a standard OpenSSH server. For SOCKS4/5 tests SSH is used to perform
- the SOCKS functionality and requires a SSH client and server.
-
- The base port number (8990), which all the individual port numbers are
- indexed from, can be set explicitly using runtests.pl' -b option to allow
- running more than one instance of the test suite simultaneously on one
- machine, or just move the servers in case you have local services on any of
- those ports.
-
- The HTTP server supports listening on a Unix domain socket, the default
- location is 'http.sock'.
-
- 1.4 Run
-
- './configure && make && make test'. This builds the test suite support code
- and invokes the 'runtests.pl' perl script to run all the tests. Edit the top
- variables of that script in case you have some specific needs, or run the
- script manually (after the support code has been built).
-
- The script breaks on the first test that doesn't do OK. Use -a to prevent
- the script from aborting on the first error. Run the script with -v for more
- verbose output. Use -d to run the test servers with debug output enabled as
- well. Specifying -k keeps all the log files generated by the test intact.
-
- Use -s for shorter output, or pass test numbers to run specific tests only
- (like "./runtests.pl 3 4" to test 3 and 4 only). It also supports test case
- ranges with 'to', as in "./runtests 3 to 9" which runs the seven tests from
- 3 to 9. Any test numbers starting with ! are disabled, as are any test
- numbers found in the files data/DISABLED or data/DISABLED.local (one per
- line). The latter is meant for local temporary disables and will be ignored
- by git.
-
- When -s is not present, each successful test will display on one line the
- test number and description and on the next line a set of flags, the test
- result, current test sequence, total number of tests to be run and an
- estimated amount of time to complete the test run. The flags consist of
- these letters describing what is checked in this test:
-
- s stdout
- d data
- u upload
- p protocol
- o output
- e exit code
- m memory
- v valgrind
-
- 1.5 Shell startup scripts
-
- Tests which use the ssh test server, SCP/SFTP/SOCKS tests, might be badly
- influenced by the output of system wide or user specific shell startup
- scripts, .bashrc, .profile, /etc/csh.cshrc, .login, /etc/bashrc, etc. which
- output text messages or escape sequences on user login. When these shell
- startup messages or escape sequences are output they might corrupt the
- expected stream of data which flows to the sftp-server or from the ssh
- client which can result in bad test behaviour or even prevent the test
- server from running.
-
- If the test suite ssh or sftp server fails to start up and logs the message
- 'Received message too long' then you are certainly suffering the unwanted
- output of a shell startup script. Locate, cleanup or adjust the shell
- script.
-
- 1.6 Memory test
-
- The test script will check that all allocated memory is freed properly IF
- curl has been built with the CURLDEBUG define set. The script will
- automatically detect if that is the case, and it will use the
- 'memanalyze.pl' script to analyze the memory debugging output.
-
- Also, if you run tests on a machine where valgrind is found, the script will
- use valgrind to run the test with (unless you use -n) to further verify
- correctness.
-
- runtests.pl's -t option will enable torture testing mode, which runs each
- test many times and makes each different memory allocation fail on each
- successive run. This tests the out of memory error handling code to ensure
- that memory leaks do not occur even in those situations. It can help to
- compile curl with CPPFLAGS=-DMEMDEBUG_LOG_SYNC when using this option, to
- ensure that the memory log file is properly written even if curl crashes.
-
- 1.7 Debug
-
- If a test case fails, you can conveniently get the script to invoke the
- debugger (gdb) for you with the server running and the exact same command
- line parameters that failed. Just invoke 'runtests.pl <test number> -g' and
- then just type 'run' in the debugger to perform the command through the
- debugger.
-
- 1.8 Logs
-
- All logs are generated in the log/ subdirectory (it is emptied first in the
- runtests.pl script). Use runtests.pl -k to force it to keep the temporary
- files after the test run since successful runs will clean it up otherwise.
-
- 1.9 Test input files
-
- All test cases are put in the data/ subdirectory. Each test is stored in the
- file named according to the test number.
-
- See FILEFORMAT for the description of the test case files.
-
- 1.10 Code coverage
-
- gcc provides a tool that can determine the code coverage figures for
- the test suite. To use it, configure curl with
- CFLAGS='-fprofile-arcs -ftest-coverage -g -O0'. Make sure you run the normal
- and torture tests to get more full coverage, i.e. do:
-
- make test
- make test-torture
-
- The graphical tool ggcov can be used to browse the source and create
- coverage reports on *NIX hosts:
-
- ggcov -r lib src
-
- The text mode tool gcov may also be used, but it doesn't handle object files
- in more than one directory very well.
-
- 1.11 Remote testing
-
- The runtests.pl script provides some hooks to allow curl to be tested on a
- machine where perl can not be run. The test framework in this case runs on
- a workstation where perl is available, while curl itself is run on a remote
- system using ssh or some other remote execution method. See the comments at
- the beginning of runtests.pl for details.
-
-2. Numbering
-
- 2.1 Test case numbering
-
- Test cases used to be numbered by category, but the ranges filled
- up. Subsets of tests can now be selected by passing keywords to the
- runtests.pl script via the make TFLAGS variable.
-
- New tests should now be added by finding a free number in
- tests/data/Makefile.inc.
-
-3. Write tests
-
- Here's a quick description on writing test cases. We basically have three
- kinds of tests: the ones that test the curl tool, the ones that build small
- applications and test libcurl directly and the unit tests that test
- individual (possibly internal) functions.
-
- 3.1 test data
-
- Each test has a master file that controls all the test data. What to read,
- what the protocol exchange should look like, what exit code to expect and
- what command line arguments to use etc.
-
- These files are tests/data/test[num] where [num] is described in section 2
- of this document, and the XML-like file format of them is described in the
- separate tests/FILEFORMAT document.
-
- 3.2 curl tests
-
- A test case that runs the curl tool and verifies that it gets the correct
- data, it sends the correct data, it uses the correct protocol primitives
- etc.
-
- 3.3 libcurl tests
-
- The libcurl tests are identical to the curl ones, except that they use a
- specific and dedicated custom-built program to run instead of "curl". This
- tool is built from source code placed in tests/libtest and if you want to
- make a new libcurl test that is where you add your code.
-
- 3.4 unit tests
-
- Unit tests are tests in the 13xx sequence and they are placed in tests/unit.
- There's a tests/unit/README describing the specific set of checks and macros
- that may be used when writing tests that verify behaviors of specific
- individual functions.
-
- The unit tests depend on curl being built with debug enabled.
-
-4. TODO
-
- 4.1 More protocols
-
- Add tests for TELNET, LDAP, DICT...
-
- 4.2 SOCKS auth
-
- SOCKS4/5 test deficiencies - no proxy authentication tests as SSH (the
- test mechanism) doesn't support them
diff --git a/tests/README.md b/tests/README.md
new file mode 100644
index 0000000..4b524a0
--- /dev/null
+++ b/tests/README.md
@@ -0,0 +1,229 @@
+# The curl Test Suite
+
+# Running
+
+## Requires to run
+
+ - perl (and a unix-style shell)
+ - python (and a unix-style shell, for SMB and TELNET tests)
+ - python-impacket (for SMB tests)
+ - diff (when a test fails, a diff is shown)
+ - stunnel (for HTTPS and FTPS tests)
+ - OpenSSH or SunSSH (for SCP, SFTP and SOCKS4/5 tests)
+ - nghttpx (for HTTP/2 tests)
+ - nroff (for --manual tests)
+
+### Installation of python-impacket
+
+ The Python-based test servers support both recent Python 2 and 3.
+ You can figure out your default Python interpreter with python -V
+
+ Please install python-impacket in the correct Python environment.
+ You can use pip or your OS' package manager to install 'impacket'.
+
+ On Debian/Ubuntu the package names are:
+
+ - Python 2: 'python-impacket'
+ - Python 3: 'python3-impacket'
+
+ On FreeBSD the package names are:
+
+ - Python 2: 'py27-impacket'
+ - Python 3: 'py37-impacket'
+
+ On any system where pip is available:
+
+ - Python 2: 'pip2 install impacket'
+ - Python 3: 'pip3 install impacket'
+
+ You may also need to manually install the Python package 'six'
+ as that may be a missing requirement for impacket on Python 3.
+
+### Port numbers used by test servers
+
+ All test servers run on "random" port numbers. All tests should be written
+ to use suitable variables instead of fixed port numbers so that test cases
+ continue to work independent on what port numbers the test servers actually
+ use.
+
+ See [FILEFORMAT](FILEFORMAT.md) for the port number variables.
+
+### Test servers
+
+ The test suite runs simple FTP, POP3, IMAP, SMTP, HTTP and TFTP stand-alone
+ servers on the ports listed above to which it makes requests. For SSL tests,
+ it runs stunnel to handle encryption to the regular servers. For SSH, it
+ runs a standard OpenSSH server. For SOCKS4/5 tests SSH is used to perform
+ the SOCKS functionality and requires a SSH client and server.
+
+ The base port number (8990), which all the individual port numbers are
+ indexed from, can be set explicitly using runtests.pl' -b option to allow
+ running more than one instance of the test suite simultaneously on one
+ machine, or just move the servers in case you have local services on any of
+ those ports.
+
+ The HTTP server supports listening on a Unix domain socket, the default
+ location is 'http.sock'.
+
+### Run
+
+ `./configure && make && make test`. This builds the test suite support code
+ and invokes the 'runtests.pl' perl script to run all the tests. Edit the top
+ variables of that script in case you have some specific needs, or run the
+ script manually (after the support code has been built).
+
+ The script breaks on the first test that doesn't do OK. Use `-a` to prevent
+ the script from aborting on the first error. Run the script with `-v` for
+ more verbose output. Use `-d` to run the test servers with debug output
+ enabled as well. Specifying `-k` keeps all the log files generated by the
+ test intact.
+
+ Use `-s` for shorter output, or pass test numbers to run specific tests only
+ (like `./runtests.pl 3 4` to test 3 and 4 only). It also supports test case
+ ranges with 'to', as in `./runtests.pl 3 to 9` which runs the seven tests
+ from 3 to 9. Any test numbers starting with ! are disabled, as are any test
+ numbers found in the files `data/DISABLED` or `data/DISABLED.local` (one per
+ line). The latter is meant for local temporary disables and will be ignored
+ by git.
+
+ When `-s` is not present, each successful test will display on one line the
+ test number and description and on the next line a set of flags, the test
+ result, current test sequence, total number of tests to be run and an
+ estimated amount of time to complete the test run. The flags consist of
+ these letters describing what is checked in this test:
+
+ s stdout
+ d data
+ u upload
+ p protocol
+ o output
+ e exit code
+ m memory
+ v valgrind
+
+### Shell startup scripts
+
+ Tests which use the ssh test server, SCP/SFTP/SOCKS tests, might be badly
+ influenced by the output of system wide or user specific shell startup
+ scripts, .bashrc, .profile, /etc/csh.cshrc, .login, /etc/bashrc, etc. which
+ output text messages or escape sequences on user login. When these shell
+ startup messages or escape sequences are output they might corrupt the
+ expected stream of data which flows to the sftp-server or from the ssh
+ client which can result in bad test behaviour or even prevent the test
+ server from running.
+
+ If the test suite ssh or sftp server fails to start up and logs the message
+ 'Received message too long' then you are certainly suffering the unwanted
+ output of a shell startup script. Locate, cleanup or adjust the shell
+ script.
+
+### Memory test
+
+ The test script will check that all allocated memory is freed properly IF
+ curl has been built with the `CURLDEBUG` define set. The script will
+ automatically detect if that is the case, and it will use the
+ 'memanalyze.pl' script to analyze the memory debugging output.
+
+ Also, if you run tests on a machine where valgrind is found, the script will
+ use valgrind to run the test with (unless you use `-n`) to further verify
+ correctness.
+
+ runtests.pl's `-t` option will enable torture testing mode, which runs each
+ test many times and makes each different memory allocation fail on each
+ successive run. This tests the out of memory error handling code to ensure
+ that memory leaks do not occur even in those situations. It can help to
+ compile curl with `CPPFLAGS=-DMEMDEBUG_LOG_SYNC` when using this option, to
+ ensure that the memory log file is properly written even if curl crashes.
+
+### Debug
+
+ If a test case fails, you can conveniently get the script to invoke the
+ debugger (gdb) for you with the server running and the exact same command
+ line parameters that failed. Just invoke `runtests.pl <test number> -g` and
+ then just type 'run' in the debugger to perform the command through the
+ debugger.
+
+### Logs
+
+ All logs are generated in the log/ subdirectory (it is emptied first in the
+ runtests.pl script). They remain in there after a test run.
+
+### Test input files
+
+ All test cases are put in the `data/` subdirectory. Each test is stored in
+ the file named according to the test number.
+
+ See [FILEFORMAT.md](FILEFORMAT.md) for a description of the test case file
+ format.
+
+### Code coverage
+
+ gcc provides a tool that can determine the code coverage figures for the
+ test suite. To use it, configure curl with `CFLAGS='-fprofile-arcs
+ -ftest-coverage -g -O0`. Make sure you run the normal and torture tests to
+ get more full coverage, i.e. do:
+
+ make test
+ make test-torture
+
+ The graphical tool ggcov can be used to browse the source and create
+ coverage reports on *NIX hosts:
+
+ ggcov -r lib src
+
+ The text mode tool gcov may also be used, but it doesn't handle object files
+ in more than one directory very well.
+
+### Remote testing
+
+ The runtests.pl script provides some hooks to allow curl to be tested on a
+ machine where perl can not be run. The test framework in this case runs on
+ a workstation where perl is available, while curl itself is run on a remote
+ system using ssh or some other remote execution method. See the comments at
+ the beginning of runtests.pl for details.
+
+## Test case numbering
+
+ Test cases used to be numbered by category ranges, but the ranges filled
+ up. Subsets of tests can now be selected by passing keywords to the
+ runtests.pl script via the make `TFLAGS` variable.
+
+ New tests are added by finding a free number in `tests/data/Makefile.inc`.
+
+## Write tests
+
+ Here's a quick description on writing test cases. We basically have three
+ kinds of tests: the ones that test the curl tool, the ones that build small
+ applications and test libcurl directly and the unit tests that test
+ individual (possibly internal) functions.
+
+### test data
+
+ Each test has a master file that controls all the test data. What to read,
+ what the protocol exchange should look like, what exit code to expect and
+ what command line arguments to use etc.
+
+ These files are `tests/data/test[num]` where `[num]` is just a unique
+ identifier described above, and the XML-like file format of them is
+ described in the separate [FILEFORMAT.md](FILEFORMAT.md) document.
+
+### curl tests
+
+ A test case that runs the curl tool and verifies that it gets the correct
+ data, it sends the correct data, it uses the correct protocol primitives
+ etc.
+
+### libcurl tests
+
+ The libcurl tests are identical to the curl ones, except that they use a
+ specific and dedicated custom-built program to run instead of "curl". This
+ tool is built from source code placed in `tests/libtest` and if you want to
+ make a new libcurl test that is where you add your code.
+
+### unit tests
+
+ Unit tests are placed in `tests/unit`. There's a tests/unit/README
+ describing the specific set of checks and macros that may be used when
+ writing tests that verify behaviors of specific individual functions.
+
+ The unit tests depend on curl being built with debug enabled.
