Table of Contents
Introduction
In addition to definition of the messages and their contents, every communication protocol must ensure that the message is successfully delivered over the I/O link to the other side. The serialised message payload must be wrapped in some kind of transport information prior to being sent and unwrapped on the other side when received.
For example, let's define a custom protocol that wraps the message payload in the following way:
where:
- SYNC - 2 bytes of synchronisation value to indicate beginning of the message, must be "0xab 0xcd"
- SIZE - 2 bytes, length of remaining data including checksum and not including SIZE field itself.
- ID - 1 byte, numeric ID of the message.
- PAYLOAD - any number of bytes, serialised message data
- CHECKSUM - 2 bytes, CRC-CCITT value of all bytes starting (and including) from SIZE field and ending after PAYLOAD field.
The processing of the raw bytes received over I/O link involves identifying the fields listed above and stripping them off one by one until the PAYLOAD is reached, where it can be read by the created proper message object (based on read message ID). If one of the elements is not as it is expected to be, the processing should stop.
The sequential processing the the transport information values, and stripping them one by one before proceeding to the next one, may remind of OSI Conceptual Model, where a layer serves the layer above it and is served by the layer below it.
The COMMS library defines every such layer, that is handling a single value, as separate class. Every such layer class will use field abstraction (see Fields Definition Tutorial) to wrap the value it handles. The layer classes are stacked together by wrapping one another. When combined together they are called Protocol Stack.
The wrapping for the example above will look like this:
When presented as actual stack, it may look like this:
Please note that CHECKSUM layer lays between SYNC and SIZE. This is a bit counter intuitive, because SIZE follows SYNC in the protocol description, while CHECKSUM appears last. The reason for such location of CHECKSUM layer is that it calculates and verifies checksum on the SIZE, ID, and PAYLOAD areas, i.e. it must wrap the all three.
The COMMS library provides multiple classes to define various layers when assembling the full protocol stack of layers. All these classes reside in comms::protocol namespace. The following sections will cover all the layer classes required to assemble the protocol stack described above.
PAYLOAD Layer
The top layer, that is responsible to read/write the payload of the message is called PAYLOAD. It is implemented by comms::protocol::MsgDataLayer class in the COMMS library.
NOTE, that comms::protocol::MsgDataLayer receives a template parameter. In the normal operation, when transport frame fields are not stored anywhere, it is never used. However, there is way to perform read operation while caching transport fields (by using readFieldsCached()) The payload field is defined to be comms::field::ArrayList of raw data (see comms::protocol::MsgDataLayer::Field). It would be wise to provide a way to supply extra options to choose storage type for this field, when defining protocol stack. As the result the definition becomes:
ID Layer
The ID layer is responsible to process the ID of the message and based on this ID, create proper message object. The COMMS library implements this functionality in comms::protocol::MsgIdLayer class. It receives at least four template parameters. The first one is a type of the field that can be used to read/write the ID information. The Common Interface Class section described my_protocol::MsgId enum type used to define message IDs, it can be reused to define a field responsible to read / write message ID value
NOTE, that underlying enum type is defined to be std::uint8_t, which will result in 1 byte serialisation length.
The second parameter is common interface class for all input messages that need to be recognised during read operation. This type will be defined by the application and is expected to be an alias (typedef) or extending class to my_protocol::Message (described in Common Interface Class)
The third parameter is all the types of all the custom messages, that need to be recognised in read operation, bundled in std::tuple.
NOTE, that the interface class (TMessage) passed as the second parameter is expected to be the common base class for all the messages passed as third one.
The fourth template parameter is the upper layer it needs to wrap:
NOTE, that all the input messages are passed as a template parameter with a default value (bundling all the available messages). It will give an opportunity to the application to use only messages it needs.
Also note, input messages in the bundle (TInputMessages) are expected to be defined in order of their numeric IDs. It is allowed to have separate message classes to report the same numeric ID. However, the read operation will try to read all the messages with the found ID one by one in order of their definition until success is reported.
The comms::protocol::MsgIdLayer defines MsgPtr internal type, which is smart pointer (std::unique_ptr) to the input message interface class (TMessage) provided as second template parameter.
During the normal read operation, the comms::protocol::MsgIdLayer will dynamically allocate the proper message object.
The comms::protocol::MsgIdLayer can also be used in bare metal systems, that do NOT use dynamic memory allocation. In order to prevent this layer from using dynamic memory allocation, the comms::option::app::InPlaceAllocation option needs to be passed as fifth template parameter to the comms::protocol::MsgIdLayer class. However, an ability to use this option needs to be provided to the application itself only if needed. In order to achive that additional template parameter needs to be used.
In this case, the comms::protocol::MsgIdLayer will statically allocate internal buffer in its private data members, big enough to hold any message object of any type listed in AllMessages bundle. It means that only one message object can be allocated and used at a time, i.e. the previous object must be deleted prior to new one being allocated.
Also, the MsgPtr will still be a variant of std::unique_ptr, but with custom deleter (defined by COMMS library itself), which will make sure the proper destruction of the message object and release of the internal buffer for next allocation. In case new allocation is attempted when internal buffer is NOT released, the new message will NOT be allocated and read operation will fail with comms::ErrorStatus::MsgAllocFailure error.
By default, if the received data contains unknown message ID (the message type is not in AllMessages bundle), the read operation returns comms::ErrorStatus::InvalidMsgId and no message object is allocated. However, there are bridge / gateway / firewall type of applications which are interested to decode only limited number of messages, but still forward the received data (sometimes changing the transport wrapping) without actually decoding the contents. In this case the default behaviour cannot be used. The COMMS library provides comms::GenericMessage message definition which has a single variable length data field (defined using comms::field::ArrayList class). The comms::protocol::MsgIdLayer may also receive comms::option::app::SupportGenericMessage option specifying type of the GenericMessage. In this case, if the appropriate message type hasn't been found in AllMessages bundle, the appropriate comms::GenericMessage object will be created instead. However, just like with comms::option::app::InPlaceAllocation, this option should be used by the application if needed.
Note, that comms::option::app::SupportGenericMessage and comms::option::app::InPlaceAllocation options can be used together. In this case the comms::GenericMessage message object will be allocated in the same allocation area. The client application will be able to combine these option together in single tuple and use pass as TAllocationOptions parameter.
When constructed, the comms::protocol::MsgIdLayer creates an array of statically allocated factory methods, which are responsible to allocate right message objects. This array is used as a map of message ID to the factory method. The COMMS library contains inner logic that analyses a tuple of all input message types provided to comms::protocol::MsgIdLayer. If the IDs of the messages are sequential ones starting from a low number such as 0 or 1, and the highest ID value do not significantly exceed the total number of message types in the tuple, then the one-to-one mapping is generated, i.e. to access the right factory method is just accessing the right cell in the mapping array (O(1) time complexity). In all other cases the factory methods are compacted together and binary search is executed to get appropriate factory method having the numeric message ID value (O(log(n))).
NOTE, that comms::protocol::MsgIdLayer doesn't use any dynamic memory allocation to store internal factory methods, that create proper message object given the ID of the message, which makes it possible and safe to use in bare-metal environment without any HEAP.
It may happen that comms::protocol::MsgIdLayer class as-is is not really suitable for implementing message identification and creation of message object when implementing custom protocol. It is possible to implement a new custom layer (see Implementing New Layers section below) with the required functionality. However, it is recommended to use comms::MsgFactory object internally. It will help in creation the proper message object once the ID value is known.
SIZE Layer
The SIZE layer is responsible to process information on the remaining message length, and forward the read/write operations to the upper layer in case it is safe to do so. The COMMS library provides comms::protocol::MsgSizeLayer class for that purpose.
The comms::protocol::MsgSizeLayer receives at least two template parameters. The first one is the definition of the field (see Fields Definition Tutorial for details) that is responsible to read/write the remaining length information. The second template parameter is an upper layer that is being wrapped. The third template parameter is optional default behaviour modification options.
Please note the usage of comms::option::def::NumValueSerOffset option when defining the field type. If it is NOT used, the serialised length value will cover only ID and PAYLOAD (layers it wraps). However, according to the protocol specification, the SIZE value must also include CHECKSUM. Usage of comms::option::def::NumValueSerOffset <sizeof(std::uint16_t)> will add 2 (sizeof(std::uint16_t)) when serialising the length of wrapped fields. See also Serialisation Offset for more details.
CHECKSUM Layer
The CHECKSUM layer is responsible to calculate and verify the checksum on the data read and/or written by the upper layers it wraps. The COMMS library provides comms::protocol::ChecksumLayer and comms::protocol::ChecksumPrefixLayer for this purpose. They are very similar. The only difference is that comms::protocol::ChecksumLayer appends the checksum value, while comms::protocol::ChecksumPrefixLayer prepends it.
The both layer classes receives three template parameters. The first one is a field that is responsible to read/write the checksum value.
The second template parameter is a checksum calculator class which is used to calculate a checksum value. Please refer to the documentation of comms::protocol::ChecksumLayer or comms::protocol::ChecksumPrefixLayer class for the details on the interface this checksum calculator class must provide. The example above uses comms::protocol::checksum::Crc_CCITT, which calculates the the standard CRC-CCITT value. All the checksum calculators the COMMS library provides reside in comms::protocol::checksum namespace.
The third template parameter is an upper layer that is being wrapped.
By default both comms::protocol::ChecksumLayer and comms::protocol::ChecksumPrefixLayer allow inner (upper) layers to complete their read operation before calculating and verifying checksum on read data. However, there may be protocols that may allow checksum verification before attempting to read message contents. In this case comms::option::def::ChecksumLayerVerifyBeforeRead option may be used as fourth template parameter.
SYNC Layer
The SYNC layer is responsible to recognise the synchronisation byte(s) in the input stream as well as write appropriate value when the write operation takes place. The COMMS library provides comms::protocol::SyncPrefixLayer class that helps with this task.
The comms::protocol::SyncPrefixLayer class receives two template parameters. The first one is the type of the field, that is responsible to read/write the synchronisation byte(s). Please note the usage of comms::option::def::DefaultNumValue option when defining the field type. It insures that the default constructed field will have the required value.
The second template parameter is the upper layer being wrapped.
Extra Transport Values
Some protocol may use extra values as part of the transport information. Such values may have an influence on how the message payload is read and/or on how the message object is handled. As an example let's define the following transport wrapping:
The VERSION value is expected to influence the "read" operation. The message object may have some extra fields, which were introduced in later version of the protocol, and it needs to take into account the provided VERSION info.
The COMMS library provides comms::protocol::TransportValueLayer to handle such fields. HOWEVER it requires extra support from common message interface class. The latter must use comms::option::def::ExtraTransportFields option in order to define expected interface (please refer to Extra Transport Values for details).
The comms::protocol::TransportValueLayer class receives three template parameters. The first one is the field used to read / write the value. The second parameter is index of the relevant extra transport field in the comms::Message::TransportFields tuple. And the third parameter is the next layer.
The whole protocol stack definition may look like this:
NOTE, that in the example above VERSION layer follows ID. In this case the message object is already created by the ID layer when VERSION one performs its read operation. The latter may update the version information inside the created message object. However, there may be cases when extra transport value precedes ID layer:
The COMMS library is also capable of handling such case. It contains internal "magic", which forces some layers to complete their read operation and update created message object (if necessary) before the read operation is forwarded to the final (PAYLOAD) layer.
Unfortunatelly there are layers (comms::protocol::ChecksumLayer, comms::protocol::ChecksumPrefixLayer, and comms::protocol::MsgSizeLayer), that cannot complete their read operation, without read of the PAYLOAD data being complete as well. As the result these layers do not support being wrapped by comms::protocol::TransportValueLayer and will fail compilation with static assert if such wrapping is attempted.
Pseudo Transport Values
Some protocols may report one of the values (such as protocol version) in one of the messages used to establish connection. After that, the reported value may have influence on how other message contents are being read. Handling such case is very similar to Extra Transport Values. The only difference is passing comms::option::def::PseudoValue option to comms::protocol::TransportValueLayer layer class. It will cause the transport value not actually being (de)serialised during read / write operations. The pseudo field value is going to be stored as private member of comms::protocol::TransportValueLayer and can be accessed (and updated) using pseudoField() member function(s). During the read operation the comms::protocol::TransportValueLayer behaves as if the value stored in this field was actually read.
Layers Summary
The earlier examples show that layer classes wrap one another, which creates the following picture:
The outermost (or bottom) layer defines a full protocol stack. It should be typedef-ed or extended to avoid any confusion:
Every protocol layer provides an ability to access the next one using nextLayer() member function. It is strongly recommended to generate convenience access functions using COMMS_PROTOCOL_LAYERS_ACCESS() macro.
It is equivalent to having the following member function being defined:
Please note the following:
- Every provided name is prefixed with layer_.
- The names inside the COMMS_PROTOCOL_LAYERS_ACCESS() macro specify innermost layer first and outermost one last. There is a convenience wrapper macro COMMS_PROTOCOL_LAYERS_ACCESS_OUTER() which allows listing in the opposite order (outermost first and innermost last).
WARNING: Some compilers, such as clang or earlier versions of gcc (v4.9 and earlier) may have problems compiling the COMMS_PROTOCOL_LAYERS_ACCESS() macro even though it contains valid C++11 code. If the compilation failure happens there is a need to define inner Base type which specifies exact type of the protocol stack base class.
NOTE, that COMMS library also defines COMMS_MUST_DEFINE_BASE in case the base class definition is needed (going to be used). If the developed application is going to be multi-platform and compiled with various compilers (some of which may warn about unused private type) it is possible to use the defined symbol to add / remove the definition of the Base member type.
Since v5.2 the COMMS library defines COMMS_PROTOCOL_LAYERS_NAMES() and COMMS_PROTOCOL_LAYERS_NAMES_OUTER() macros they are similar to the COMMS_PROTOCOL_LAYERS_NAMES() and COMMS_PROTOCOL_LAYERS_NAMES_OUTER() respectively, but also provide aliases to the layer types. However usage of these macros requires inner Base type definition of the base class:
It is equivalent to having the following types and member function being defined:
Implementing New Layers
Every protocol is unique, and there is a chance that COMMS library doesn't provide all the necessary layer classes required to implement custom logic of the protocol. The COMMS library allows implementation and usage of custom layers as long as it defines the required types and implements required functions.
Some of the available layers support extension of their default functionality. Pleases check the following tutorial pages. In most cases it will be sufficient.
- Defining Custom Message ID Protocol Stack Layer
- Defining Custom Message Size Protocol Stack Layer
- Defining Custom Transport Value Protocol Stack Layer
- Defining Custom Checksum Protocol Stack Layer
- Defining Custom Sync Prefix Protocol Stack Layer
In case absolutely new and independent protocol stack layer needs to be implemented, please follow the instructions below.
It is strongly recommended to inherit from comms::protocol::ProtocolLayerBase and implement missing functionality
Note that the third template parameter to the comms::protocol::ProtocolLayerBase base class is the inheriting class itself.
The comms::protocol::ProtocolLayerBase implements read() and readFieldsCached() member functions which are actual "read" interface, they invoke the doRead() member function implemented the derived layer class, while providing the "nextLayerReader" object to be used to forward the read operation to the next layer. The signature of the nextLayerReader.read() function is the same as read(). NOTE, that msg parameter to the doRead() member function can be either reference to a smart pointer (MsgPtr) or a reference to the message object itself (one that extends comms::MessageBase). If the message object contents need to be accessed, then it is necessary to know what exactly is passed as msg parameter to the doRead() function. The isMessageObjRef() member function can be used to help in such task and "tag dispatch idiom" can be used to perform right functionality.
In similar way comms::protocol::ProtocolLayerBase implements write() and writeFieldsCached() member functions which are actual "write" interface, they invoke the doWrite() member function implemented the derived layer class, while providing the "nextLayerWriter" object to be used to forward the read operation to the next layer. The signature of the nextLayerWriter.write() function is the same as write(). If the doWrite() function requires modification to the used iterator (moving it back and force), it must determine type of the iterators (using std::iterator_traits). In case the used iterator is output one (not random-access), then such update may be impossible. In this case the doWrite() function is expected to write some dummy value and return comms::ErrorStatus::UpdateRequired. It will indicate to the client application that invocation of update functionality with random access iterator needs to follow the write operation.
The comms::protocol::ProtocolLayerBase base class defines also "update" interface functions update() and updateFieldsCached(), which in the similar way invoke doUpdate(). However, comms::protocol::ProtocolLayerBase class provides a default implementation of doUpdate() member function, which does nothing, just advances the iterator. If there is a need for a custom update functionality, please provide it in you layer class by implementing the custom version of doUpdate() member function.
The signature of the nextLayerUpdater.update() function is the same as update().
There may be cases when update functionality requires also knowledge about message object being written. In this case there is a need to provide overloaded doUpdate() member function that receives message object as its first parameter.
If the new layer being implemented is similar to comms::protocol::MsgIdLayer, i.e. creates message objects when id of the message is known, then it must also override (hide) the inherited createMsg() and implement its own version:
