Page MenuHomePhabricator

Mbed TLS continuous integration
Updated 119 Days AgoPublic

All code that is included in Mbed TLS must be contributed in the form of pull requests on GitHub and undergoes some automated testing. This page describes the continuous integration jobs that run on every pull request.


This job checks that all commits have a Signed-off-by: line. The presence of this line indicates that the author of the commit certifies that the commit is covered by the Developer Certificate of Origin and contributed according to the project license.

All commits must have such a line, otherwise the commit cannot be accepted for legal reasons. As a temporary exception, commits from Arm employees or from other contributors who already have a contributor license agreement (CLA) can still be accepted, but please include a Signed-off-by: line in any new work.

If the DCO job fails, please reword all commit messages that are missing a Signed-off-by: line. If you have multiple commit messages to rewrite, How to use git interactive rebase for signing off a series of commits may help.


The Travis job runs a set of sanity checks and test runs on x86. You can find the list of tests in .travis.yml. The Travis logs are public.

PR-head and PR-merge jobs

The PR-NNN-head and PR-NNN-merge jobs run an extensive battery of tests on several platforms. The -head jobs run the tests on the tip of the submitted code. The -merge jobs run the tests on a merge with the target branch.

As of early 2020, these jobs run on a private Jenkins instance. It is unfortunately not possible to make the logs public. Please ping an Mbed TLS development team member if you need to see failure logs.

High-level overview of the Jenkins test coverage

The Jenkins PR job includes the following parts:

  • Run tests/scripts/ on Ubuntu 16.04 x86 (64-bit). This script includes:
    • Some sanity checks.
    • Tests of the library in various configurations. The tests are mainly unit tests (make test), SSL feature tests (tests/ and interoperability tests (tests/
    • Some cross compilation with GCC-arm, Arm Compiler 5 (armcc), Arm Compiler 6 (armclang) and MinGW (i686-w64-mingw32-gcc). These are only builds, not tests.
  • Run a subset of tests/scripts/ on FreeBSD (amd64)
  • Build with IAR 8 for Arm. (Temporarily omitted in Q2 2020.)
  • Build on Windows with MinGW and Visual Studio. We use the following Visual Studio versions:
    • Mbed TLS 2.7, Mbed TLS 2.16: VS 2010, 2013, 2015, 2017.
    • Since Mbed TLS 2.19: VS 2013, 2015, 2017.

Identifying Jenkins failures on a pull request

On GitHub, for jobs that ran since 2020-09-29, the statuses “PR-NNN-head TLS Testing” and “PR-NNN-merge TLS Testing” report which components failed.

Unfortunately, due to a GitHub limitation, the list is truncated to 140 characters. (It may be further truncated for display; hover over the status to see the full 140 characters.) However, if many jobs fail, it's usually a good idea to fix one and try again since failures often have a common cause. If you need the full list, or if you need logs, please post a comment to ask.

The component names are:

  • all_sh-platform-component: tests/scripts/ -k component run on platform.
  • code-coverage: tests/scripts/ run on Linux (currently Ubuntu 16.04).
  • win32-mingw, win32_msvc12_32, win32-msvc12_64: ad hoc Windows jobs in the default configuration, see below.
  • Windows-20NN: builds with the specified version of Visual Studio (Visual Studio 10 2010, Visual Studio 12 2013, Visual Studio 14 2015 or Visual Studio 15 2017).
    • Library configuration: full ( full in older branches) minus MBEDTLS_THREADING_xxx.
    • Visual Studio configurations: Release and Debug.
    • Architectures: Win32 and x64 (except no x64 in VS 2010).
    • PlatformToolset property:
      • “Retargeted” build: set to the version corresponding to the Visual Studio version for “retargeted” builds (not done for 2010).
      • “Non-retargeted” build: set to v120 on development (C99 code base), to Windows7.1SDK on 2.16 and 2.7 (C89 code base).
    • Testing: selftest.exe, plus the unit tests in the CMake builds.

The win32-mingw component runs the following bat script:

cmake . -G "MinGW Makefiles" -DCMAKE_C_COMPILER="gcc"
mingw32-make test
ctest -VV

The win32_msvc12_32 component runs the following bat script:

call "C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\vcvarsall.bat"
cmake . -G "Visual Studio 12"
MSBuild ALL_BUILD.vcxproj

The win32-msvc12_64 component is identical except that it runs cmake . -G "Visual Studio 12 Win64".


Tooling for

To run tests/scripts/, you need at least the following tools:

  • basic Unix/POSIX shell utilities. Currently any modern sh should do, but a dependency on bash may be added in 2020.
  • Git. requires a Git checkout. The following files may be restored to their checked-in version: include/mbedtls/config.h, **/Makefile. is composed of many components, some of which have additional requirements. Most components require the following tools:

  • gcc or clang (depending on the component) for native builds.
  • GNU make or CMake (depending on the component).
  • Perl 5.
  • Python ≥3.4 for Mbed TLS ≥2.20.
  • The commands openssl, gnutls-cli and gnutls-serv must be present in the $PATH to run any test component (otherwise refuses to start). If you don't have them, you can still run some test components that don't do any interoperability testing with env OPENSSL=false GNUTLS_CLI=false GNUTLS_SERV=false tests/scripts/ ….

The following table lists all the tools used by and where applicable reference versions and usage notes. Unless otherwise mentioned, more recent versions or slightly older versions should work.

GCC5.4 buildsgcc and ar must be in $PATH
GCC -m325.4 test_m32_o0 test_m32_o1See Ubuntu note below
GCC -mx325.4 test_mx32See Ubuntu note below
Clang3.8 buildsclang and ar must be in $PATH
GNU Make4.1 make will not work.
CMake3.5.1 should work
OpenSSL legacy1.0.1j test SSLv3, and TLS ciphersuites using RC4 or DESSet at $OPENSSL_LEGACY.
OpenSSL established1.0.2g TLS testingSet as $OPENSSL
OpenSSL next1.0.1j test TLS ciphersuites with ARIA or ChaChaPolySet as $OPENSSL_NEXT
Libnettle legacy2.7.1 build GnuTLS legacy
Libnettle established3.2 build GnuTLS established
Libnettle next3.4.1 build GnuTLS next
GnuTLS legacy3.3.8 test TLS ciphersuites using RC4 or DESSet as $GNUTLS_LEGACY_CLI and $GNUTLS_LEGACY_SERV
GnuTLS established3.4.10 TLS testingSet as $GNUTLS_CLI and $GNUTLS_SERV. See Ubuntu note below.
GnuTLS next3.6.5 test some DTLS featuresOptional. Set as $GNUTLS_NEXT_CLI and $GNUTLS_NEXT_SERV.
arm-none-eabi-gcc5.4-2016q3 'build_arm_none_eabi_gcc*'
MinGW-w645.3.1 build_mingw
Arm Compiler 55.06 build_armccRequires a paid license. Optional.
Arm Compiler 66.6 build_armccRequires a paid license. Optional.
Zlib1.2.8 test_zlib_cmake test_zlib_make
Perl5.22 scripts
Python3.4 scripts
Pylint2.4 check_python_filesOlder versions tend to work. Newer versions may not work due to added checks.
Doxygen1.8.11 check_doxygen_warningsRecent versions are known to fail due to tightened up warnings. 1.8.13 is ok.
Graphviz2.38.0 check_doxygen_warnings
Valgrind3.11.0 test_valgrind

Running on Ubuntu

Our reference Linux platform is Ubuntu 16.04 x86 64-bit. More recent Ubuntu versions should also work.

On Ubuntu 16.04, the following packaged tools are known to cause problems with

  • The gnutls-bin package on Ubuntu 16.04 does not work with our test scripts because Ubuntu has removed some obsolete features.

On Ubuntu 20.04, the following packaged tools are known to cause problems with

  • check_doxygen_warnings fails. Doxygen 1.8.11 and 1.8.13 are known to work.

On Ubuntu, for the 32-bit builds, install the following packages:

apt-get install gcc-multilib libc6-dev-i386 linux-libc-dev:i386 libc6:386

Instead of gcc-multilib, which conflicts with cross-compilers for non-Intel architectures, a versioned package (e.g. gcc5-multilib) will do, but you need to create an additional symbolic link:

ln -s x86_64-linux-gnu/asm /usr/include/asm

Notes about networking

The tests attempt to establish an IPv6 connection to the local host. This is optional, and will be skipped if IPv6 is not available. However, if IPv6 is possible but IPv6 connections to localhost are blocked by a firewall, the test will fail.

Docker environment

A Dockerfile is provided in tests/docker. Please note that this Dockerfile is not actively maintained and may not be suitable to run all of We plan to publish the Dockerfiles used on our CI systems in 2020.

Windows tooling


Last Author
Last Edited
Sep 30 2020, 9:17 AM

Event Timeline

gilles-peskine-arm edited the content of this document. (Show Details)Jul 16 2020, 8:52 PM
gilles-peskine-arm edited the content of this document. (Show Details)Jul 16 2020, 9:42 PM
gilles-peskine-arm edited the content of this document. (Show Details)Sep 28 2020, 8:01 PM
gilles-peskine-arm edited the content of this document. (Show Details)Sep 30 2020, 9:17 AM