All code that is included in Mbed TLS must be contributed in the form of [[https://github.com/ARMmbed/mbedtls/pulls | pull requests on GitHub]] and undergoes some automated testing. This page describes the continuous integration jobs that run on every pull request.
== DCO ==
This job checks that all commits have a `Signed-off-by:` line. The presence of this line indicates that the author of the commit certifies that the commit is covered by the [[https://github.com/ARMmbed/mbedtls/blob/development/dco.txt | Developer Certificate of Origin]] and contributed according to the [[https://github.com/ARMmbed/mbedtls/blob/development/README.md#License | project license]].
All commits must have such a line, otherwise the commit cannot be accepted for legal reasons. As a temporary exception, commits from Arm employees or from other contributors who already have a contributor license agreement (CLA) can still be accepted, but please include a `Signed-off-by:` line in any new work.
If the DCO job fails, please reword all commit messages that are missing a `Signed-off-by:` line. If you have multiple commit messages to rewrite, [[https://stackoverflow.com/questions/25570947/how-to-use-git-interactive-rebase-for-signing-off-a-series-of-commits | How to use git interactive rebase for signing off a series of commits]] may help.
== Travis ==
The Travis job runs a set of sanity checks and test runs on x86. You can find the list of tests in [[https://github.com/ARMmbed/mbedtls/blob/development/.travis.yml | .travis.yml]]. The Travis logs are public.
== PR-head and PR-merge jobs ==
The PR-//NNN//-head and PR-//NNN//-merge jobs run an extensive battery of tests on several platforms. The -head jobs run the tests on the tip of the submitted code. The -merge jobs run the tests on a merge with the target branch.
As of early 2020, these jobs run on a private Jenkins instance. It is unfortunately not possible to make the logs public. Please ping an Mbed TLS development team member if you need to see failure logs.
=== High-level overview of the Jenkins test coverage ===
The Jenkins PR job includes the following parts:
- Run `tests/scripts/all.sh` on Ubuntu 16.04 x86 (64-bit). This script includes:
- Some sanity checks.
- Tests of the library in various configurations. The tests are mainly unit tests (`make test`), SSL feature tests (`tests/ssl-opt.sh`) and interoperability tests (`tests/compat.sh`).
- Some cross compilation with GCC-arm, Arm Compiler 5 (`armcc`), Arm Compiler 6 (`armclang`) and MinGW (`i686-w64-mingw32-gcc`). These are only builds, not tests.
- Run a subset of `tests/scripts/all.sh` on FreeBSD (amd64)
- Build with IAR 8 for Arm. (Temporarily omitted in Q2 2020.)
- Build on Windows with MinGW and Visual Studio. We use the following Visual Studio versions:
- Mbed TLS 2.7, Mbed TLS 2.16: VS 2010, 2013, 2015, 2017.
- Since Mbed TLS 2.19: VS 2013, 2015, 2017.
=== Identifying Jenkins failures on a pull request ===
On GitHub, for jobs that ran since 2020-09-29, the statuses “PR-//NNN//-head TLS Testing” and “PR-//NNN//-merge TLS Testing” report which components failed.
Unfortunately, due to a GitHub limitation, the list is truncated to 140 characters. (It may be further truncated for display; hover over the status to see the full 140 characters.) However, if many jobs fail, it's usually a good idea to fix one and try again since failures often have a common cause. If you need the full list, or if you need logs, please post a comment to ask.
The component names are:
* all_sh-//platform//-//component//: `tests/scripts/all.sh -k `//component// run on //platform//.
* `code-coverage`: `tests/scripts/basic-build-test.sh` run on Linux (currently Ubuntu 16.04).
* `win32-mingw`, `win32_msvc12_32`, `win32-msvc12_64`: ad hoc Windows jobs in the default configuration, see below.
* `Windows-`//20NN//: builds with the specified version of Visual Studio (Visual Studio 10 2010, Visual Studio 12 2013, Visual Studio 14 2015 or Visual Studio 15 2017).
* Library configuration: `config.py full` (`config.pl full` in older branches) minus `MBEDTLS_THREADING_xxx`.
* Visual Studio configurations: `Release` and `Debug`.
* Architectures: `Win32` and `x64` (except no `x64` in VS 2010).
* `PlatformToolset` property:
* “Retargeted” build: set to the version corresponding to the Visual Studio version for “retargeted” builds (not done for 2010).
* “Non-retargeted” build: set to `v120` on `development` (C99 code base), to `Windows7.1SDK` on 2.16 and 2.7 (C89 code base).
* Testing: `selftest.exe`, plus the unit tests in the CMake builds.
The `win32-mingw` component runs the following bat script:
```
cmake . -G "MinGW Makefiles" -DCMAKE_C_COMPILER="gcc"
mingw32-make
mingw32-make test
ctest -VV
programs\test\selftest.exe
```
The `win32_msvc12_32` component runs the following bat script:
```
call "C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\vcvarsall.bat"
cmake . -G "Visual Studio 12"
MSBuild ALL_BUILD.vcxproj
programs\test\Debug\selftest.exe
```
The `win32-msvc12_64` component is identical except that it runs `cmake . -G "Visual Studio 12 Win64"`.
== Tooling ==
=== Tooling for all.sh ===
To run `tests/scripts/all.sh`, you need at least the following tools:
* basic Unix/POSIX shell utilities. Currently any modern `sh` should do, but a dependency on `bash` may be added in 2020.
* Git. `all.sh` requires a Git checkout. The following files may be restored to their checked-in version: `include/mbedtls/config.h`, `**/Makefile`.
`all.sh` is composed of many components, some of which have additional requirements. Most components require the following tools:
* `gcc` or `clang` (depending on the component) for native builds.
* GNU make or CMake (depending on the component).
* Perl 5.
* Python ≥3.4 for Mbed TLS ≥2.20.
* The commands `openssl`, `gnutls-cli` and `gnutls-serv` must be present in the `$PATH` to run any test component (otherwise `all.sh` refuses to start). If you don't have them, you can still run some test components that don't do any interoperability testing with `env OPENSSL=false GNUTLS_CLI=false GNUTLS_SERV=false tests/scripts/all.sh …`.
The following table lists all the tools used by `all.sh` and where applicable reference versions and usage notes. Unless otherwise mentioned, more recent versions or slightly older versions should work.
| Tool | Version | Source | Usage | Notes |
| ---- | ------- | ------ | ----- | ----- |
| Git | 2.7.4 | https://git-scm.com/downloads | | |
| GCC | 5.4 | https://gcc.gnu.org/ | Native builds | `gcc` and `ar` must be in `$PATH` |
| GCC `-m32` | 5.4 | https://gcc.gnu.org/ | For `all.sh test_m32_o0 test_m32_o1` | See Ubuntu note below |
| GCC `-mx32` | 5.4 | https://gcc.gnu.org/ | For `all.sh test_mx32` | See Ubuntu note below |
| Clang | 3.8 | https://clang.llvm.org/ | Native builds | `clang` and `ar` must be in `$PATH` |
| GNU Make | 4.1 | https://www.gnu.org/software/make/ | | Non-GNU make will not work. |
| CMake | 3.5.1 | https://cmake.org/ | | 2.8 should work |
| OpenSSL legacy | 1.0.1j | https://www.openssl.org/source/ | To test SSLv3, and TLS ciphersuites using RC4 or DES | Set at `$OPENSSL_LEGACY`. |
| OpenSSL established | 1.0.2g | https://www.openssl.org/source/ | For TLS testing | Set as `$OPENSSL` |
| OpenSSL next | 1.0.1j | https://www.openssl.org/source/ | To test TLS ciphersuites with ARIA or ChaChaPoly | Set as `$OPENSSL_NEXT` |
| Libnettle legacy | 2.7.1 | http://www.lysator.liu.se/~nisse/nettle/ | To build GnuTLS legacy | |
| Libnettle established | 3.2 | http://www.lysator.liu.se/~nisse/nettle/ | To build GnuTLS established | |
| Libnettle next | 3.4.1 | http://www.lysator.liu.se/~nisse/nettle/ | To build GnuTLS next | |
| GnuTLS legacy | 3.3.8 | https://gnutls.org/download.html | To test TLS ciphersuites using RC4 or DES | Set as `$GNUTLS_LEGACY_CLI` and `$GNUTLS_LEGACY_SERV` |
| GnuTLS established | 3.4.10 | https://gnutls.org/download.html | For TLS testing | Set as `$GNUTLS_CLI` and `$GNUTLS_SERV`. See Ubuntu note below. |
| GnuTLS next | 3.6.5 | https://gnutls.org/download.html | To test some DTLS features | Optional. Set as `$GNUTLS_NEXT_CLI` and `$GNUTLS_NEXT_SERV`. |
| arm-none-eabi-gcc | 5.4-2016q3 | https://gcc.gnu.org/ | For `all.sh 'build_arm_none_eabi_gcc*'` | |
| MinGW-w64 | 5.3.1 | http://mingw-w64.org/ | For `all.sh build_mingw` | |
| Arm Compiler 5 | 5.06 | https://developer.arm.com/tools-and-software/embedded/arm-compiler/arm-compiler-5/downloads | For `all.sh build_armcc` | Requires a paid license. Optional. |
| Arm Compiler 6 | 6.6 | https://developer.arm.com/tools-and-software/embedded/arm-compiler/downloads/version-6 | For `all.sh build_armcc` | Requires a paid license. Optional. |
| Zlib | 1.2.8 | https://zlib.net/ | For `all.sh test_zlib_cmake test_zlib_make` | |
| Perl | 5.22 | https://www.perl.org/get.html | Test scripts | |
| Python | 3.4 | https://www.python.org/downloads/ | Test scripts | |
| Pylint | 2.4 | https://pypi.org/project/pylint/ | For `all.sh check_python_files` | Older versions tend to work. Newer versions may not work due to added checks. |
| Doxygen | 1.8.11 | https://www.doxygen.nl/download.html | For `all.sh check_doxygen_warnings` | [Recent versions are known to fail due to tightened up warnings](https://github.com/ARMmbed/mbedtls/issues/3497). 1.8.13 is ok. |
| Graphviz | 2.38.0 | https://graphviz.org/download/ | For `all.sh check_doxygen_warnings` | |
| Valgrind | 3.11.0 | https://valgrind.org/ | For `all.sh test_valgrind` | |
=== Running `all.sh` on Ubuntu
Our reference Linux platform is Ubuntu 16.04 x86 64-bit. More recent Ubuntu versions should also work.
On Ubuntu 16.04, the following packaged tools are known to cause problems with `all.sh`:
* The `gnutls-bin` package on Ubuntu 16.04 does not work with our test scripts because Ubuntu has removed some obsolete features.
On Ubuntu 20.04, the following packaged tools are known to cause problems with `all.sh`:
* `all.sh check_doxygen_warnings` fails. Doxygen 1.8.11 and 1.8.13 are known to work.
On Ubuntu, for the 32-bit builds, install the following packages:
```
apt-get install gcc-multilib libc6-dev-i386 linux-libc-dev:i386 libc6:386
```
Instead of `gcc-multilib`, which conflicts with cross-compilers for non-Intel architectures, a versioned package (e.g. `gcc5-multilib`) will do, but you need to create an additional symbolic link:
```
ln -s x86_64-linux-gnu/asm /usr/include/asm
```
=== Notes about networking ===
The tests attempt to establish an IPv6 connection to the local host. This is optional, and will be skipped if IPv6 is not available. However, if IPv6 is possible but IPv6 connections to localhost are blocked by a firewall, the test will fail.
=== Docker environment ===
A Dockerfile is provided in [tests/docker](https://github.com/ARMmbed/mbedtls/tree/development/tests/docker). Please note that this Dockerfile is not actively maintained and may not be suitable to run all of `all.sh`. We plan to publish the Dockerfiles used on our CI systems in 2020.
=== Windows tooling ===
TODO