Fast Message Producer module "fmproducer"
Description
Almost all data sources in process technology transmit data in packed data fields consisting of bytes. The content and, in some cases, the length of these packets can be identified via a messageKey (or "ID"). Examples are provided in the following table.
| Protocol | Message Key | Key Length |
|---|---|---|
| CAN Bus | CAN ID | 11 or 29 bits |
| MVB Bus | Port Number | 12 bits |
| ProfiBus | Slave Node ID | 7 bits |
| Modbus | Register, Coil, Input, Status | 2 + 16 bits |
Various smartCORE plugins (e.g., fmudp, canbus, smartmvb) implement the connection to the hardware via dedicated interfaces, receive or send data packets, and feed the received packets into the so-called Fast Message Dispatcher (FMD). This provides an efficient technology within smartCORE for reading measurement data and status information from any data source according to a recurring pattern and producing it into individual smartCORE channels (hence the name "fm Producer").
The interface modules create at least one instance of an FMD under their own name. This is connected in the fmProducer via the fmd property.
In addition to the MessageID and a timestamp, the received packets contain data in a data field consisting of bytes. For a channel, the data can be found starting at a specific bitOffset with a fixed bitLength. The byte order (byteOrder) in the buffer, as well as (unfortunately) the addressSpec(ification), which determines how the bits in the buffer are counted, also play a role.
After the raw data bytes have been put in the correct order, the value has been correctly shifted and masked, interpretation takes place via the imageType. Does the number have a sign or not? Should the number be read as BCD (Binary Coded Decimal) or as a polar value (fixed-point)? Integer, floating-point, or even text?
After optional scaling to a physical measurement unit (scale, offset, physicalUnit) according to
the data value is output to the smartCORE channel along with the (receive) timestamp, which is also transmitted by the interface module.
Interfaces & Protocols Used
- Fast Message Dispatching
JSON Configuration
The following section describes the module’s entire JSON configuration and explains the individual parameters.
Example Configuration (Minimal)
{
"module":"FmProducer",
"factory":"fmproducer",
"config":{
"fmd":"FastMessageDispatcher",
"channels":[
{
"name":"Channel",
"messageKey":42,
"bitOffset":0,
"bitLength":32,
"imageType":"float"
},
[...]
]
}
}
Sample configuration (maximum)
{
"module":"FmProducerSmartMVB",
"factory":"fmproducer",
"config":{
"fmd":"smartmvb0",
"bufferSize":1024,
"namespace":["directory","subDirectory"],
"channelPrefix":"SmartMVB",
"addressSpec": "EN61375_MVB",
"byteOrder": "bigEndian",
"channels":[
{
"name":"SmartMVBmessageData",
"bufferSize":1024,
"scale":3.14,
"offset":2.72,
"physicalDimension":"Length",
"physicalUnit":"m",
"messageKey":10012,
"bitOffset":0,
"bitLength":20,
"imageType":"unsigned",
"addressSpec": "AscendingFirstBit",
"byteOrder": "littleEndian",
"absoluteTolerance":0.5
},
[...]
]
}
}
Global Module Parameters
| Parameter Name | Required | Data Type | Valid Range | Default | Description |
|---|---|---|---|---|---|
| fmd | YES | STRING | Fast Message Dispatcher of the sending module | ||
| bufferSize | No1 | INT | 1 - | 1024 | (default) Buffer size of the created channels |
| channelPrefix | No | STRING | Channel prefix (useful when multiple fmproducer modules are used and the configurations have overlapping channel names) | ||
| namespace | No | ARRAY [STRING] | Channel prefix in the form of hierarchically concatenated namespaces | ||
| addressSpec | No | STRING | "EN61375_MVB" | Default addressing scheme | |
| byteOrder | No | STRING | "BigEndian" | Default byte order | |
| channels | Yes | JSON Array | List of JSON objects for configured channels |
Configuration of a Channel (JSON Object)
| Parameter Name | Required | Data Type | Valid Value Range | Default | Description |
|---|---|---|---|---|---|
| Filter | |||||
| messageKey | No2 | UINT | ID of the received message (e.g., CAN bus message ID, MVB port, ...) | ||
| Process Image | |||||
| addressSpec | No | STRING | (Value from global setting) | Addressing scheme, see description below | |
| byteOrder | No | STRING | "BigEndian", "LittleEndian", "[A-Z]+" | (Value from global setting) | Byte order of the stored value, see description below |
| imageType | YES | STRING | Supported source data type of the stored value (see below) | ||
| bitOffset | YES | UINT | Bit offset of the value stored in the message | ||
| bitLength | No3 | UINT | Bit length of the value stored in the message | ||
| stringHint | No4 | STRING | Hint for determining the string length (see below) | ||
| stringEncoding5 | No4 | STRING | IANA name, e.g. "UTF-8", "windows-1252" | 6 | Character encoding of the string in the data packet (see below) |
| timeFix5 | No7 | STRING | "RFC4330", "RFC5905" | "RFC4330" | Handling of the time64 rollover in 2036 (see below) |
| scale | No8 | FLOAT | 1 | Scaling factor of the produced value | |
| offset | No8 | FLOAT | 0 | Additive offset of the produced value | |
| smartCORE | |||||
| name | YES | STRING | Name of the channel | ||
| dataType | No9 | STRING | Data type of the channel | ||
| bufferSize | No1 | INT | 1 - | 1024 | Buffer size of the created channel |
| physicalDimension | No | STRING | Physical dimension | ||
| physicalUnit | No | STRING | Physical unit | ||
| noFilter | No | BOOL | false | Disables the DataReduction filter | |
| absoluteTolerance | No | FLOAT | 0.0 | Absolute tolerance for the DataReduction filter | |
| cacheSize | No10 | INT | 0 | Number of data samples stored in a local cache of the channel before they are published to other modules. | |
| Reserved | |||||
| debug | No | BOOL | false | Enables debug output for the channel. | |
| numElements | No | INT | 1 | Length of a data field in multiples of the base element |
Address specifications 'addressSpec'
Why must an addressing scheme be defined?
The fmproducer parses the data packet received from an external device. The specifications of this device and the interface used determine how individual data channels are to be located within the data stream. Unfortunately, there are many variations here because, despite standardization in certain areas, there are still plenty of proprietary implementations.
The address specification addressSpec determines the counting method used to specify the MSBit or LSBit in bitOffset. For historical reasons (fmProducer was originally developed for MVB), the setting EN61375_MVB is also the default setting.
Configurable Values
Unless otherwise specified, the position of the first bit (MSB/LSB depending on byteOrder) is addressed. Specifying addressSpec has no effect if a free mapping is selected as byteOrder.
| Enum | Alias | Description |
|---|---|---|
| EN61375_MVB | MVB EN61375 RevNum ReversedNum ReversedForMsbNumerics | Strictly follows EN61375 with regard to byte order, data types, alignment, and bit counting. - The bitOffset is byteOffset * 8 + rightShift - Non-compliant configurations are rejected. |
| ReversedForMsb | Rev Reversed | 11 For all signals, bits are counted in reverse (Motorola) order. |
| AscendingFirstBit | DBC | For all signals, bits are counted in ascending order of significance. |
| AscendingLSBit | positionLsb | For all signals, bits are counted in ascending order of significance. The position of the LSBit is always addressed. |
| AscendingMSBit | positionMsb | For all signals, bits are counted in ascending order. The position of the MSBit is always addressed. |
Determination of the Image Position for EN61375_MVB
In the EN 61375-2-1 standard, starting from §6.4.2, there are various guidelines on how data must be transmitted on the MVB. Certain cases are excluded from the outset. Channels with a non-compliant configuration are logged as errors in the log file in fmproducer starting with smartCORE 2.10.1 and blocked from processing.
-
(Assumption and gray area)12 The standard does not define data types that do not have a length that is an exact power of 2. A numeric value with another bitLength is processed, but reported with a warning in the log file at startup. Exempt are imageTypes whose length is defined by the type itself (e.g. timedate48).
-
Thus, bitOffset does not consistently refer to the MSBit or LSBit of a variable, but rather either to the first (integer) byte or the LSBit in a byte for
-
The representation is always big-endian. This excludes little-endian arrangements.
-
The image position must always be aligned to a multiple of its size in the data buffer. For data types with a length of 1, 2, or 4 bits, these can therefore only lie within a single byte. All others with lengths of 8, 16, 32, or 64 bits must be aligned to a byte. Shifting the image is not permitted! => if .
-
A value below 8 bits begins and ends in the same byte. This also applies to a bitLength that is not a power of 2: a 3-bit value with crosses the byte boundary and is rejected.
A value below 8 bits that crosses the byte boundary is rejected. A bitLength that is not a power of 2 is reported.
If decoding using this EN61375_MVB scheme is not possible, one of the other schemes can still be selected for a single channel, and then shifting and realigning the bits and bytes can be attempted.
Counting Bit Positions
The most important distinction for all other schemes is, first and foremost, the order in which the bits are counted.
With the Ascending* settings, the bits are counted in ascending order of value. This is typical for CANbus or Profibus configurations and also corresponds to implementations in programming languages.
With the ReversedForMsb setting, the bits in byteOrder BigEndian (Motorola, MSB, Network) are strictly counted from left to right. This corresponds to the counting method at the physical level in serial data transmission protocols, such as CANbus, SPI, or MVB.
The graphic illustrates the different counting methods for 3 bytes as an example.
Defining the anchor bit for the image position
In the Ascending* settings, a distinction is made as to which bit of the data value is addressed by bitOffset. Usually, it is the first bit (AscendingFirstBit), i.e., the MSBit for byteOrder bigEndian and the LSBit for littleEndian. However, there are also exceptions where, regardless of the byteOrder, either the MSBit or the LSBit is always addressed.
In the following diagram, two 18-bit integer values are extracted from a data telegram with Ascending* counting, one transmitted with byteOrder littleEndian and one with bigEndian.
And this diagram shows an 18-bit integer value in ReversedForMsb counting:
The table lists possible addressing variants for the int18 values shown above.
| addressSpec | byteOrder | bitOffset | bitLength |
|---|---|---|---|
| ReversedForMsb AscendingFirstBit AscendingLSBit | LittleEndian | 11 | 18 |
| AscendingMSBit | LittleEndian | 28 | 18 |
| (no function) | "CBA" | 1113 | 18 |
| ReversedForMsb | BigEndian | 13 | 18 |
| AscendingFirstBit AscendingMSBit | BigEndian | 34 | 18 |
| AscendingLSBit | BigEndian | 49 | 18 |
| (no function) | "ABC" | 3314 | 18 |
Boolean Signals
Last but not least, for Boolean signals, the counting direction alone determines the selected bit:
| addressSpec | byteOrder | bitOffset | bitLength |
|---|---|---|---|
| ReversedForMsb | BigEndian | 13 | 1 |
| (all others) | BigEndian LittleEndian | 10 | 1 |
| (no function) | "A" | 1015 | 1 |
Byte order 'byteOrder'
| Enum | Alias | Description |
|---|---|---|
| BigEndian | Big, MSB, Motorola, Network | The byte with the most significant bit (MSBit) is at the beginning of the transmitted data packet |
| LittleEndian | Little, LSB, Intel | The byte with the least significant bit (LSBit) is at the beginning of the transmitted data packet |
| Sequence from A-Z | If a mapping string is used, the bytes can be rearranged from any jumbled order into the correct sequence. addressSpec is therefore redundant. |
Using a mapping string
The bytes in the receive buffer are indexed starting at bitOffset / 8, beginning with 'A' in ascending order. The order of the bytes in descending order of significance is determined by the string. bitOffset MODULO 8 determines by how many bits the extracted value must be shifted to the right so that the least significant bit (LSB) is at position .
With this flexible mapping, it is generally possible to interpret all telegrams that have traveled a long distance from a measurement terminal, through bus couplers, controllers, gateways, etc., and have undergone various reordering and interpretations of the byte sequence along the way. Every system offers its own configuration options, and these are used extensively. It could be so simple...
Example:
bytes: 0_______ 1_______ 2_______ 3_______ 4_______ 5_______ 6_______
bits: 76543210 76543210 76543210 76543210 76543210 76543210 76543210
bitOffset: --------------------->|
Indexing: A B C D E ...
Extraction for "CDBA"
value = ((((byte[4] << 8) // alias 'C' (bitOffset / 8) + 2
| byte[5] << 8) // alias 'D' (bitOffset / 8) + 3
| byte[3] << 8) // alias 'B' (bitOffset / 8) + 1
| byte[2]) // alias 'A' (bitOffset / 8) + 0
>> (bitOffset % 8);
Source data types 'imageType'
| Enum | Alias | Description |
|---|---|---|
| bool | boolean | Fixed bitLength of 1 and dataType Bool. |
| unsigned | Unsigned integer with bitLength 2..64. Can be calculated with scale and offset to form a scaled physical quantity. | |
| antivalent, antivalent2 | => unsigned with bitLength = 2, mostly used in the MVB context for safe Boolean values: 0: ERROR 1: FALSE 2: TRUE 3: UNDEFINED | |
| signed | Signed integer with bitLength 2..64; the most significant bit is the sign bit. Can be calculated with scale and offset to form a scaled physical quantity. | |
| bcd | Binary-coded decimal numbers, 4 bits each for representing a digit 0..9 | |
| float | 16 Fixed bitLength of 32, IEEE 754 | |
| double | 16 Fixed bitLength of 64, IEEE 754 | |
| timedate48 | Fixed bitLength of 48, EN 61375-2-1 §6.4.6.2 (TCN, WTB, MVB), see Timestamps | |
| time64 | Fixed bitLength of 64, RFC 1305, see Timestamps | |
| bytearray | Additional properties define how the length of the array is calculated | |
| string | Additional properties define how the length of the string is calculated | |
UniPolar<M>.<N> | Unsigned fixed-point number with <M> integer bits and a fixed bitLength of <N> bits.Can be converted to a scaled physical quantity using scale and offset. EN 61375-2-1 §6.4.3.7 (TCN, WTB, MVB) | |
BiPolar<M>.<N> | Signed fixed-point number with <M> integer bits (including sign) and a fixed bitLength of <N> bits.Can be calculated with scale and offset to yield a scaled physical quantity. EN 61375-2-1 §6.4.3.8 (TCN, WTB, MVB) |
Timestamps 'timedate48' and 'time64'
time64 is counted from 1900 as RFC 1305 defines it; the value used to be 70 years off. A timestamp of 0 is passed on as 0.
Both imageTypes are produced as nanoseconds since 01.01.1970 (UTC) with dataType int64. The conversion uses integer arithmetic; the fraction of a second is rounded to the nearest nanosecond.
| imageType | Structure | Reference | valid until |
|---|---|---|---|
| timedate48 | 32 bit seconds (unsigned) + 16 bit fraction in s | 01.01.1970 | 07.02.2106 |
| time64 | 32 bit seconds (unsigned) + 32 bit fraction in s | 01.01.1900 | see timeFix |
A timestamp whose bits are all 0 is passed on as 0 and thus marks a missing timestamp.
Rollover of time64 in 2036 'timeFix'
The seconds counter of time64 rolls over on 07.02.2036 at 06:28:16 UTC. timeFix determines which era a received value refers to:
| timeFix | Method |
|---|---|
| RFC4330 (default) | RFC 4330 §3: If the most significant bit of the seconds is set, the time lies between 1968 and 2036, otherwise between 2036 and 2104. Fixed mapping without reference to the system time. |
| RFC5905 | RFC 5905 §6: The era closest to the receive timestamp applies. Requires a correct system time. |
If a timestamp deviates by more than 30 years from the receive timestamp, this is reported once per channel in the log file with both points in time. A typical cause is a remote station that sends seconds since 1970 instead of since 1900.
Notes on determining the length of a string 'stringHint' and 'numElements'
- Every string ends at the first null character, regardless of the stringHint.
- bitOffset and bitLength of string and bytearray must be multiples of 8; addressSpec and byteOrder have no effect on them.
- Without addressSpec (default EN61375_MVB), strings are read as ISO-8859-1.
"stringEncoding": "UTF-8"restores the previous behaviour.
Strings and data fields (imageType string and bytearray) are always located byte by byte in the data packet:
-
bitOffset and bitLength must be multiples of 8. Otherwise the channel is disabled at startup with a message in the log file.
-
bitOffset is the byte position × 8. addressSpec and byteOrder have no effect; byteOrder only determines the byte order of a length field (see below).
The area reserved for the string in the data buffer is determined by
-
the bitLength,
-
numElements × bitLength if numElements is specified. bitLength is then the width of one character (default 8 bit, 16 bit for U8WLength to U32WLength and S7WLength), or
-
without either, the rest of the message from bitOffset on.
Regardless of the stringHint, every string ends at the first null character (U+0000) after decoding. Whatever follows in the reserved area is discarded. If all bytes are needed, use imageType bytearray.
| Enum | Alias | Description |
|---|---|---|
| FixedLengthInBits | bitLength | (default) The string fills the reserved area. |
| NullTerminated | Equivalent to FixedLengthInBits, kept for compatibility | |
| FixedLength | numElements defines the number of characters in the string. | |
| U8Length | A uint8 ahead of the text gives its length in bytes. | |
| U16Length | A uint16 ahead of the text gives its length in bytes. | |
| U32Length | A uint32 ahead of the text gives its length in bytes. | |
| U8WLength5 | A uint8 ahead of the text gives its length in words (2 bytes, e.g. UTF-16). | |
| U16WLength5 | A uint16 ahead of the text gives its length in words. | |
| U32WLength5 | A uint32 ahead of the text gives its length in words. | |
| S7Length5 | SIMATIC STRING, see below | |
| S7WLength5 | SIMATIC WSTRING, see below | |
| EndOfMessage | The string always extends to the end of the message. |
Strings with a length field
With U8Length to U32WLength, a length field precedes the text at bitOffset. The reserved width from bitLength or numElements applies to the text; the length field comes on top. A length field of 2 or 4 bytes is read little endian with byteOrder LittleEndian, otherwise big endian.
If the length field states more than the reserved area or the message contains, the text is cut and this is reported once per channel in the log file. If the length field is missing from the message, no value is produced.
Example U16Length, byteOrder BigEndian, bitOffset 0:
00 02 41 42 43 → "AB"
SIMATIC strings 'S7Length' and 'S7WLength'
SIMATIC controllers store strings with a header of declared maximum length and current length. Both count characters. Only the characters of the current length are read; whatever follows in the reserved area is irrelevant.
| stringHint | SIMATIC data type | Header | Character | Default stringEncoding | Maximum length |
|---|---|---|---|---|---|
| S7Length | STRING[n] | 1 byte maximum length, 1 byte current length | 1 byte | windows-1252 | 254 |
| S7WLength | WSTRING[n] | 2 bytes maximum length, 2 bytes current length, big endian | 2 bytes | UTF-16BE | 16382 |
numElements corresponds to the declaration n and is optional; without it, the rest of the message applies. The header of S7WLength is always big endian, regardless of byteOrder.
If the current length exceeds the maximum length, it is cut to the maximum length. This, as well as a maximum length above the limit or differing from numElements, is reported once per channel in the log file; the value is read anyway.
Example STRING[10] := 'abcdef' with numElements 10:
0A 06 61 62 63 64 65 66 xx xx xx xx → "abcdef"
Character encoding 'stringEncoding'
Internally, smartCORE processes all strings as Unicode. stringEncoding specifies the encoding of the bytes in the data packet. If not specified, "windows-1252" applies for S7Length, "UTF-16BE" for S7WLength, "ISO-8859-1" (Latin-1) for addressSpec EN61375_MVB, otherwise "UTF-8".
Permitted are the names and aliases from the IANA registry "Character Sets". Upper and lower case do not matter. The aliases with prefix cs (e.g. csUTF8) are not supported. An unknown name is reported in the log file at startup and the channel is disabled.
Frequently used encodings and the bytes of some characters:
| stringEncoding | ä | Ä | ß | € | ° | µ | ² |
|---|---|---|---|---|---|---|---|
| UTF-8 | C3 A4 | C3 84 | C3 9F | E2 82 AC | C2 B0 | C2 B5 | C2 B2 |
| ISO-8859-1 (Latin-1) | E4 | C4 | DF | – | B0 | B5 | B2 |
| ISO-8859-15 (Latin-9) | E4 | C4 | DF | A4 | B0 | B5 | B2 |
| windows-1252 | E4 | C4 | DF | 80 | B0 | B5 | B2 |
| IBM437 (cp437) | 84 | 8E | E1 | – | F8 | E6 | FD |
| UTF-16BE (MVB UNICODE16) | 00 E4 | 00 C4 | 00 DF | 20 AC | 00 B0 | 00 B5 | 00 B2 |
| US-ASCII | – | – | – | – | – | – | – |
US-ASCII only knows the bytes 00 to 7F. A character that does not exist in the encoding, or an invalid byte sequence, appears in the channel as the replacement character "�" (U+FFFD).
Each message is decoded on its own. A byte sequence that is cut off at the end of the data area is not joined with the next message, but is also output as "�".
Typical error patterns with a wrong setting:
| Shown in channel | Cause |
|---|---|
ä instead of ä | The data is UTF-8, the setting is ISO-8859-1 or windows-1252. |
� instead of ä | The data is ISO-8859-1 or windows-1252, the setting is UTF-8. |
„ instead of ä | The data is IBM437, the setting is windows-1252. |
Notes on determining the length of a 'numElements' data field
The byte alignment of bitOffset and bitLength applies to data fields as well (see above).
imageType: ByteArray
The function for decoding data fields has not yet been verified. Use only after consultation!
Channel data types (target data types) 'dataType'
The dataType is always automatically determined from the imageType if it is not specified. A target format is set to prevent information loss while minimizing memory usage.
| Enum | Alias | Value range | Application |
|---|---|---|---|
| Boolean | bool | false/true | Status/control signals |
| Float | up to | Measurement data | |
| Double | up to | ||
| Integer8 | int8 | ||
| Integer16 | int16 | ||
| Integer32 | int32 | ||
| Integer64 | int64 | Timestamp | |
| UnsignedInteger8 | uint8 | Status Codes | |
| UnsignedInteger16 | uint16 | ||
| UnsignedInteger32 | uint32 | ||
| UnsignedInteger64 | uint64 | ||
| ByteArray | |||
| String | Text |
Enforced data types:
If bitLength == 1 or imageType == bool, then dataType is automatically always bool.
For imageType == string or bytearray, the dataType is always string or bytearray.
The imageTypes time64 or timedate48 enforce the dataType as int64 to correctly and completely convert timestamps to nanoseconds since January 1, 1970.
Specifying numElements > 1 forces the dataType to ByteArray, unless an imageType of String is set.
A numeric imageType cannot fill a dataType String, ByteArray or GpsLocation; such a channel is disabled at start with a message naming both types.5
Automatic data types:
In this case, the floating-point formats float and double take precedence, provided one of the following criteria is met:
- the data source already provides a floating-point or fixed-point value via the imagetype (float, double, unipolar, bipolar),
- or ,
- a physicalUnit is specified.
If , a float is sufficient for lossless conversion.
Integer formats are also selected based on the bitLength and the presence of a sign (signed) in the imageType for the smallest possible data range:
- (Unsigned)Integer8 for
- (Unsigned)Integer16 for
- (Unsigned)Integer32 for
- (Unsigned)Integer64 for
The optional sign is padded from the most significant bit of the image to the width of the dataType.
Manual Data Types and Conversion:
Manually selecting the dataType can result in a loss of precision, resolution, or a significant restriction of the available value range and should therefore generally be avoided!
If the data source returns a floating-point number according to the aforementioned rules, it is rounded for conversion to an integer dataType: from .50 up to the next larger integer value.
In any case, the value ranges of the selected integer types are taken into account. If the value from the image lies outside the available range, the corresponding minimum or maximum that can be represented is substituted, and a message is logged.
Data Reduction Filter
A data reduction filter is configured for each channel. By default, this ensures that new data records are only written to the channel when the data content changes (OnChange). The data’s reception timestamp, however, is continuously updated with each processed data record, enabling the (constant) data to be processed in other plugins up to the most recent point in time.
The absoluteTolerance option sets a tolerance band around the last written value. A new data record is only published if the difference from the last written value exceeds this tolerance band. This setting is useful for highly noisy signals to significantly reduce the data volume.
If the noFilter option is set to true, no data reduction filter is created. This setting is useful for fast data that must not be reduced, for example, because vibration signals are subsequently analyzed in the frequency domain.
Module Information
| Information | Value |
|---|---|
| Authors | optiMEAS GmbH |
| Since smartCORE | 0.103 |
| Module Type | Fast Message Receiver, Producer |
| Dependencies | Fast Message Sender Module (e.g., fmudp, canbus, smartmvb, rawplayback, ...) |