Mbed TLS continuous integration

Updated 1,543 Days AgoPublic

Actions

All code that is included in Mbed TLS must be contributed in the form of pull requests on GitHub and undergoes some automated testing. This page describes the continuous integration jobs that run on every pull request.

DCO

This job checks that all commits have a Signed-off-by: line. The presence of this line indicates that the author of the commit certifies that the commit is covered by the Developer Certificate of Origin and contributed according to the project license.

All commits must have such a line, otherwise the commit cannot be accepted for legal reasons. As a temporary exception, commits from Arm employees or from other contributors who already have a contributor license agreement (CLA) can still be accepted, but please include a Signed-off-by: line in any new work.

If the DCO job fails, please reword all commit messages that are missing a Signed-off-by: line. If you have multiple commit messages to rewrite, How to use git interactive rebase for signing off a series of commits may help.

Travis

The Travis job runs a set of sanity checks and test runs on x86. You can find the list of tests in .travis.yml. The Travis logs are public.

PR-head and PR-merge jobs

The PR-NNN-head and PR-NNN-merge jobs run an extensive battery of tests on several platforms. The -head jobs run the tests on the tip of the submitted code. The -merge jobs run the tests on a merge with the target branch.

As of early 2020, these jobs run on a private Jenkins instance. It is unfortunately not possible to make the logs public. Please ping an Mbed TLS development team member if you need to see failure logs.

High-level overview of the Jenkins test coverage

The Jenkins PR job includes the following parts:

• Run tests/scripts/all.sh on Ubuntu 16.04 x86 (64-bit). This script includes:
  • Some sanity checks.
  • Tests of the library in various configurations. The tests are mainly unit tests (make test), SSL feature tests (tests/ssl-opt.sh) and interoperability tests (tests/compat.sh).
  • Some cross compilation with GCC-arm, Arm Compiler 5 (armcc), Arm Compiler 6 (armclang) and MinGW (i686-w64-mingw32-gcc). These are only builds, not tests.
• Run a subset of tests/scripts/all.sh on FreeBSD (amd64)
• Build with IAR 8 for Arm. (Temporarily omitted in Q2 2020.)
• Build on Windows with MinGW and Visual Studio. We use the following Visual Studio versions:
  • Mbed TLS 2.7, Mbed TLS 2.16: VS 2010, 2013, 2015, 2017.
  • Since Mbed TLS 2.19: VS 2013, 2015, 2017.

Identifying Jenkins failures on a pull request

On GitHub, for jobs that ran since 2020-09-29, the statuses “PR-NNN-head TLS Testing” and “PR-NNN-merge TLS Testing” report which components failed.

Unfortunately, due to a GitHub limitation, the list is truncated to 140 characters. (It may be further truncated for display; hover over the status to see the full 140 characters.) However, if many jobs fail, it's usually a good idea to fix one and try again since failures often have a common cause. If you need the full list, or if you need logs, please post a comment to ask.

The component names are:

• all_sh-platform-component: tests/scripts/all.sh -k component run on platform.
• code-coverage: tests/scripts/basic-build-test.sh run on Linux (currently Ubuntu 16.04).
• win32-mingw, win32_msvc12_32, win32-msvc12_64: ad hoc Windows jobs in the default configuration, see below.
• Windows-20NN: builds with the specified version of Visual Studio (Visual Studio 10 2010, Visual Studio 12 2013, Visual Studio 14 2015 or Visual Studio 15 2017).
  • Library configuration: config.py full (config.pl full in older branches) minus MBEDTLS_THREADING_xxx.
  • Visual Studio configurations: Release and Debug.
  • Architectures: Win32 and x64 (except no x64 in VS 2010).
  • PlatformToolset property:
    • “Retargeted” build: set to the version corresponding to the Visual Studio version for “retargeted” builds (not done for 2010).
    • “Non-retargeted” build: set to v120 on development (C99 code base), to Windows7.1SDK on 2.16 and 2.7 (C89 code base).
  • Testing: selftest.exe, plus the unit tests in the CMake builds.

The win32-mingw component runs the following bat script:

cmake . -G "MinGW Makefiles" -DCMAKE_C_COMPILER="gcc"
mingw32-make
mingw32-make test
ctest -VV
programs\test\selftest.exe

The win32_msvc12_32 component runs the following bat script:

call "C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\vcvarsall.bat"
cmake . -G "Visual Studio 12"
MSBuild ALL_BUILD.vcxproj
programs\test\Debug\selftest.exe

The win32-msvc12_64 component is identical except that it runs cmake . -G "Visual Studio 12 Win64".

Tooling

Tooling for all.sh

To run tests/scripts/all.sh, you need at least the following tools:

• basic Unix/POSIX shell utilities. Currently any modern sh should do, but a dependency on bash may be added in 2020.
• Git. all.sh requires a Git checkout. The following files may be restored to their checked-in version: include/mbedtls/config.h, **/Makefile.

all.sh is composed of many components, some of which have additional requirements. Most components require the following tools:

• gcc or clang (depending on the component) for native builds.
• GNU make or CMake (depending on the component).
• Perl 5.
• Python ≥3.4 for Mbed TLS ≥2.20.
• The commands openssl, gnutls-cli and gnutls-serv must be present in the $PATH to run any test component (otherwise all.sh refuses to start). If you don't have them, you can still run some test components that don't do any interoperability testing with env OPENSSL=false GNUTLS_CLI=false GNUTLS_SERV=false tests/scripts/all.sh ….

The following table lists all the tools used by all.sh and where applicable reference versions and usage notes. Unless otherwise mentioned, more recent versions or slightly older versions should work.

Running all.sh on Ubuntu

Our reference Linux platform is Ubuntu 16.04 x86 64-bit. More recent Ubuntu versions should also work.

On Ubuntu 16.04, the following packaged tools are known to cause problems with all.sh:

• The gnutls-bin package on Ubuntu 16.04 does not work with our test scripts because Ubuntu has removed some obsolete features.

On Ubuntu 20.04, the following packaged tools are known to cause problems with all.sh:

• all.sh check_doxygen_warnings fails. Doxygen 1.8.11 and 1.8.13 are known to work.

On Ubuntu, for the 32-bit builds, install the following packages:

apt-get install gcc-multilib libc6-dev-i386 linux-libc-dev:i386 libc6:386

Instead of gcc-multilib, which conflicts with cross-compilers for non-Intel architectures, a versioned package (e.g. gcc5-multilib) will do, but you need to create an additional symbolic link:

ln -s x86_64-linux-gnu/asm /usr/include/asm

Notes about networking

The tests attempt to establish an IPv6 connection to the local host. This is optional, and will be skipped if IPv6 is not available. However, if IPv6 is possible but IPv6 connections to localhost are blocked by a firewall, the test will fail.

Docker environment

A Dockerfile is provided in tests/docker. Please note that this Dockerfile is not actively maintained and may not be suitable to run all of all.sh. We plan to publish the Dockerfiles used on our CI systems in 2020.

Windows tooling

TODO