(Fuzz) Testing of Generated Protocol Code

This article describes how to (fuzz) test the code generated by commsdsl2comms code generator. It contains the same information as the Testing Generated Protocol Code wiki page.

Overview

The commsdsl2comms utility generates a full CMake project. Since v1.4 it also generates test application which can be used to test the generated protocol code. Fuzz testing with AFL is also supported.

Let’s take a look at cc.demo1.generated repository as an example. It is generated by the cc.demo1.commsdsl project and contains test folder containing test application(s).

At this moment there is only one available application called <proj_name>_input_test. It reads the binary data from standard input and prints to standard output detected messages and values of their fields in human-readable form. The AFL fuzzer can be used to generate fuzzed input for the application.

Please open the main CMakeLists.txt file. It should list all the available options and configuration variables. Below are the options / variables relevant to the test application(s).

option (OPT_BUILD_TEST "Build and install test applications." OFF)
...

# OPT_TEST_OPTIONS - Class name of the options for test applications, defaults to demo1::options::DefaultOptions.
# OPT_TEST_INTERFACE - Class name of the interface for test applications, defaults to demo1::Message.
# OPT_TEST_FRAME - Class name of the frame for test applications, defaults to demo1::frame::Frame.
# OPT_TEST_INPUT_MESSAGES - All input messages bundle for test applications, defaults to demo1::input::AllMessages.

In order to enable the build of the test application(s) there is a need to enable OPT_BUILD_TEST option in CMake configuration.

Linux Environment

$> cd /generated/project/dir
$> mkdir build.test && cd build.test
$> cmake -DCMAKE_INSTALL_PREFIX=${PWD}/install -DOPT_BUILD_TEST=ON ..
$> make install

Windows Environment

$> cd C:\generated\project\dir
$> mkdir build.test && cd build.test
$> cmake -G "NMake Makefiles" \
     -DCMAKE_INSTALL_PREFIX=%cd%/install -DOPT_BUILD_TEST=ON ..
$> nmake install

It is recommended to use non-release build configuration to have all the assert() checks being active.

In case of AFL fuzz testing please use appropriate settings for instrumenting compilers:

$> CC=afl-gcc CXX=afl-g++ cmake -DCMAKE_INSTALL_PREFIX=${PWD}/install -DOPT_BUILD_TEST=ON ..

Extra variables allow changing the default configuration of the test application(s).

Choosing Custom Options

Please open the options folder. By default the DefaultOptions configuration is used. The generated code also contains BareMetalDefaultOptions configuration forces protocol definition NOT to use dynamic memory allocation and substitutes some problematic storage types such as std::string or std::vector to the comms::util::StaticString or comms::util::StaticVector equivalents provided by the COMMS library itself. Build configuration to use BareMetalDefaultOptions may look like this.

Linux Environment

$> cd /generated/project/dir
$> mkdir build.test && cd build.test
$> cmake -DCMAKE_INSTALL_PREFIX=${PWD}/install -DOPT_BUILD_TEST=ON \
     -DOPT_TEST_OPTIONS=test1::options::BareMetalDefaultOptions ..
$> make install

Windows Environment

$> cd C:\generated\project\dir
$> mkdir build.test && cd build.test
$> cmake -G "NMake Makefiles" -DCMAKE_INSTALL_PREFIX=%cd%/install 
    -DOPT_TEST_OPTIONS=test1::options::BareMetalDefaultOptions -DOPT_BUILD_TEST=ON ..
$> nmake install

Choosing Custom Messages

By default the test application supports all the messages of the protocol (see AllMessages). In case the protocol definition it too big (contains hundreds of messages), the compilation process of the test application may demand a lot of system resources. In such case it is recommended to test a subset of message classes. For example, the input folder contains also definition of ClientInputMessages and ServerInputMessages. They can be used to test a particular communication endpoint.

Build configuration to use ClientInputMessages may look like this.

Linux Environment

$> cd /generated/project/dir
$> mkdir build.test && cd build.test
$> cmake -DCMAKE_INSTALL_PREFIX=${PWD}/install -DOPT_BUILD_TEST=ON \
     -DOPT_TEST_INPUT_MESSAGES=test1::input::ClientInputMessages ..
$> make install

Windows Environment

$> cd C:\generated\project\dir
$> mkdir build.test && cd build.test
$> cmake -G "NMake Makefiles" -DCMAKE_INSTALL_PREFIX=%cd%/install 
    -DOPT_TEST_INPUT_MESSAGES=test1::input::ClientInputMessages -DOPT_BUILD_TEST=ON ..
$> nmake install

Other configuration

In case the protocol schema defines multiple interfaces and or multiple transport frames, please use OPT_TEST_INTERFACE and/or OPT_TEST_FRAME configuration variable to select desired interfaces class as well as transport framing.

More on AFL Fuzz Testing

The AFL requires having initial input to work with. The simplest way is to create a single binary file with random input (even single 0 byte will do).

$> printf "\x00" > ./input/zero.bin
$> afl-fuzz -i ./input -o ./output ./demo1_input_test

It may take several minutes to AFL but eventually it will be able to find binary data that leads to actual messages and exercising real messages and fields serialization.