Fast Message Producer Module "fmproducer"

Description

Almost all data sources in process technology transmit data in packed data fields of bytes. The content and possibly also the length of these packets can be identified via a messageKey (or "ID"). Examples are given in the following table.

Protocol	Message Key	Length of Key
CAN bus	CAN ID	11 or 29 bits
MVB bus	Port number	12 bits
ProfiBus	Slave node ID	7 bits
Modbus	Register, coil, input, status	2 + 16 bits

Various smartCORE plugins (e.g., fmudp, canbus, smartmvb) connect the hardware via dedicated interfaces, receive or send data packets, and feed the received packets to the so-called Fast Message Dispatcher (FMD). This provides an efficient technology in smartCORE for reading measurement data and status information from any data source according to a recurring pattern and producing it in individual smartCORE channels (hence the name "fm Producer").

The interface modules generate at least one instance of an FMD under their own name. This is connected in fmProducer via the fmd property.

fm Producer

In addition to the MessageID and a timestamp, the received packets contain the data in a data field consisting of bytes. For a channel, the data can be found in this field starting at a specific bitOffset with a defined bitLength. The arrangement of the bytes byteOrder in the buffer also plays a role, as does (unfortunately) the addressSpec(ification), which determines how the bits are counted in the buffer.

Once the raw data bytes have been put in the correct order and the value has been correctly shifted and masked, interpretation takes place via the imageType. Does the number have a sign or not? Is the number to be read as BCD (binary coded decimal) or as a polar value (fixed point)? Integer, floating point, or even text?

After optional scaling to a physical measurement (scale, offset, physicalUnit) according to

y_{phys} = y_{raw} \cdot \rm{scale} + \rm{offset}

the data value is produced in the smartCORE channel with the (receive) timestamp also transmitted by the interface module.

Interfaces & protocols used

Fast Message Dispatching

JSON configuration

The following section describes the entire JSON configuration of the module and explains the individual parameters.

Example configuration (minimal)

    {
      "module":"FmProducer",
      "factory":"fmproducer",
      "config":{
        "fmd":"FastMessageDispatcher",
        "channels":[
          {
            "name":"Channel",
            "messageKey":42,
            "bitOffset":0,
            "bitLength":32,
            "imageType":"float"
          },

          [...]

        ]
      }
    }

Example configuration (maximum)

    {
      "module":"FmProducerSmartMVB",
      "factory":"fmproducer",
      "config":{
        "fmd":"smartmvb0",
        "bufferSize":1024,
        "namespace":["directory","subDirectory"],
        "channelPrefix":"SmartMVB",
        "addressSpec": "ReversedForMsbNumerics",
        "byteOrder": "bigEndian",
        "channels":[
          {
            "name":"SmartMVBmessageData",
            "bufferSize":1024,
            "scale":3.14,
            "offset":2.72,
            "physicalDimension":"Length",
            "physicalUnit":"m",
            "messageKey":10012,
            "bitOffset":0,
            "bitLength":20,
            "imageType":"unsigned",
            "addressSpec": "AscendingMSBit",
            "byteOrder": "littleEndian",
            "absoluteTolerance":0.5
          },

          [...]

        ]
      }
    }

Global module parameters

Parameter name	Required	Data type	Meaningful value range	Default	Description
fmd	YES	STRING			Fast Message Dispatcher of the transmission module
bufferSize	No¹	INT	1 -	1024	(default) Buffer size of the created channels
channelPrefix	No	STRING			Channel prefix (useful if multiple fmproducer modules are used and the configurations have overlapping channel names)
namespace	No	ARRAY [STRING]			Channel prefix in the form of hierarchically linked namespaces
addressSpec	No	STRING		"ReversedForMsbNumerics"	Default addressing scheme
byteOrder	No	STRING		"BigEndian"	Default byte order
channels	YES	JSON Array			List of JSON objects of configured channels

Configuration of a channel (JSON object)

Parameter name	Required	Data type	Valid value range	Default	Description
					Filter
messageKey	No²	UINT			ID of the received message (e.g., CAN bus message ID, MVB port, ...)
					Process Image
addressSpec	No	STRING		(Value from global setting)	Addressing scheme, see description below
byteOrder	No	STRING	"BigEndian", "LittleEndian", "[A-Z]+"	(Value from global setting)	Byte order of the stored value, see description below
bitOffset	YES	UINT			Bit offset of the value stored in the message
bitLength	No³	UINT			Bit length of the value stored in the message
imageType	YES	STRING			Supported source data type of the stored value (see below)
stringHint	No⁴	STRING			Note on determining the string length (see below)
scale	No⁵	FLOAT		1	Scaling factor of the produced value
offset	No⁵	FLOAT		0	Additive offset of the produced value

					smartCORE
name	YES	STRING			Name of the channel
dataType	No⁶	STRING			Data type of the channel
bufferSize	No¹	INT	1 -	1024	Buffer size of the created channel
physicalDimension	No	STRING			Physical size
physicalUnit	No	STRING			Physical unit
noFilter	No	BOOL		false	Disables the DataReduction filter
absoluteTolerance	No	FLOAT		0.0	Absolute tolerance for the DataReduction filter
cacheSize	No⁷	INT		0	Number of data samples stored in a local cache of the channel before they are published to other modules.
					Reserved
debug	No	BOOL		false	Enables debug output for the channel.
numElements	No	INT		1	Length of a data field in multiples of the basic element

Supported address specifications 'addressSpec'

The address specification addressSpec determines the counting method used to specify the MSBit or LSBit in bitOffset. For historical reasons (fmProducer was first developed for MVB), the setting ReversedForMsbNumerics is also the default setting.

Unless otherwise specified, the position of the first bit (Msb/Lsb depending on byteOrder) is addressed.

The addressSpec specification has no function if a free mapping is selected as byteOrder.

Enum	Alias	Description
ReversedForMsbNumerics	MVB	⁸ For Boolean signals, the bits are counted in ascending order of value. For all other signals, bits are counted in reverse (Motorola) order.
ReversedForMsb		⁸ For all signals, bits are counted in reverse (Motorola) order.
AscendingFirstBit	DBC	For all signals, bits are counted in ascending order.
AscendingLSBit		Bits are counted in ascending order of value for all signals. The position of the LSBit is always addressed.
AscendingMSBit		Bits are counted in ascending order of value for all signals. The position of the MSBit is always addressed.

Counting the bit position

The most important distinction is initially the order in which the bits are counted.

With the Ascending* settings, the bits are counted in ascending order of value. This is typical for CAN or Profibus configurations.

With the ReversedForMsb* settings, the bits in the byteOrder BigEndian (Motorola, MSB, Network) are counted strictly from left to right. To add to the confusion, however, TCN, WTB, or MVB (EN 61375-2-1 §6.4.4.3) count bit fields in ascending order again, so the reverse counting method applies only to numerical values (ReversedForMsbNumerics), but not to individual bits.

The graphic shows the different counting methods for 3 bytes as an example.

Counting method for bits

Determining the anchor bit

The Ascending* settings also differentiate between which bit of the data value is addressed with bitOffset. Usually, this is the first bit (AscendingFirstBit), i.e., the MSBit for byteOrder bigEndian and the LSBit for littleEndian. However, there are exceptions where the MSBit or LSBit is always addressed, regardless of the byteOrder.

In the following diagram, two 18-bit integer values are extracted from a data telegram with Ascending* counting, which are transmitted once with byteOrder littleEndian and once with bigEndian.

Decoding numerical value Ascending

And this diagram supplements an 18-bit integer value in ReversedForMsb counting:

Decoding numerical value reversed

The table shows possible addressing variants for the int18 values shown above.

addressSpec	byteOrder	bitOffset	bitLength
ReversedForMsbNumerics ReversedForMsb AscendingFirstBit AscendingLSBit	LittleEndian	11	18
AscendingMSBit	LittleEndian	28	18
(no function)	"CBA"	11⁹	18
AscendingFirstBit AscendingMSBit	BigEndian	34	18
AscendingLSBit	BigEndian	49	18
(no function)	"ABC"	33¹⁰	18
ReversedForMsbNumerics ReversedForMsb	BigEndian	13	18

Boolean signals

Last but not least, with Boolean signals, the counting direction alone determines the selected bit:

Decoding Boolean

addressSpec	byteOrder	bitOffset	bitLength
ReversedForMsb	BigEndian	13	1
(all others)	BigEndian LittleEndian	10	1
(no function)	"A"	10¹¹	1

Supported byte order 'byteOrder'

Enum	Alias	Description
BigEndian	Big, MSB, Motorola, Network	The byte with the most significant bit (MSBit) is at the front of the transmitted data packet.
LittleEndian	Little, LSB, Intel	The byte with the least significant bit (LSBit) is at the front of the transmitted data packet.
Sequence from A-Z		If a mapping string is used, the bytes can be put into the correct order from any mixed arrangement. addressSpec is therefore without function.

Use of a mapping string

The bytes in the receive buffer are indexed ascending starting with 'A' from bitOffset / 8. The order of the bytes in descending order of value is determined by the string. bitOffset MODULO 8 determines how many bits the extracted value must be shifted to the right so that the least significant bit (LSBit) is at position $2^0$ .

This free mapping can generally be used to interpret all telegrams that have traveled a long way from a measuring terminal, through bus couplers, controllers, gateways, etc., and have undergone various rearrangements and interpretations of the byte order. Each system offers its own configuration options, and these are used extensively. It could be so simple...

Example:

bytes:         0_______ 1_______ 2_______ 3_______ 4_______ 5_______ 6_______
bits:          76543210 76543210 76543210 76543210 76543210 76543210 76543210
bitOffset:     --------------------->|
Indexing:                     A        B        C        D        E ...

Extraction for "CDBA"
value = ((((byte[4] << 8)       // alias 'C' (bitOffset / 8) + 2
          | byte[5] << 8)       // alias 'D' (bitOffset / 8) + 3
          | byte[3] << 8)       // alias 'B' (bitOffset / 8) + 1
          | byte[2])            // alias 'A' (bitOffset / 8) + 0
          >> (bitOffset % 8);

Supported source data types 'imageType'

Enum	Alias	Description
bool	boolean	Fixed bitLength of 1 and dataType Bool.
unsigned		Unsigned integer with bitLength 2..64. Can be calculated with scale and offset to a scaled physical quantity.
signed		Signed integer with bitLength 2..64, the most significant bit is the sign bit. Can be calculated with scale and offset to form a scaled physical quantity.
bcd		Binary-coded decimal numbers, 4 bits each for representing a digit 0..9
float		Fixed bitLength of 32, IEEE 754
double		Fixed bitLength of 64, IEEE 754
timedate48		Fixed bitLength of 48, EN 61375-2-1 §6.4.6.2 (TCN, WTB, MVB)
time64		Fixed bitLength of 64, RFC 1305
bytearray		Additional properties specify how the length of the array is calculated
string		Additional properties specify how the length of the string is calculated.
`UniPolar<M>.<N>`		Unsigned fixed-point number with `<M>` integer bits and a fixed bitLength of `<N>` bits. Can be converted to a scaled physical quantity using scale and offset. EN 61375-2-1 §6.4.3.7 (TCN, WTB, MVB)
`BiPolar<M>.<N>`		Signed fixed-point number with `<M>` integer bits (including sign) and a fixed bitLength of `<N>` bits. Can be calculated with scale and offset to obtain a scaled physical quantity. EN 61375-2-1 §6.4.3.8 (TCN, WTB, MVB)

Notes on determining the length of a string 'stringHint' and 'numElements'

warning

imageType: String
The function for decoding strings has not yet been verified. Use only after consultation!

The area reserved for the string in the data buffer is calculated from

the bitLength in integer multiples of 8 or
the specification of numElements as the number of characters.

Enum	Alias	Description
FixedLengthInBits	bitLength	bitLength / 8 defines the fixed number of characters in the string
NullTerminated		The first null character (0x00) or the end of the data range determines the end of the string
FixedLength		numElements defines the fixed number of characters in the string
U8Length		The string begins with a uint8 that dynamically specifies the length of the string.
U16Length		The string begins with a uint16 that dynamically specifies the length of the string.
U32Length		The string begins with a uint32 that dynamically specifies the length of the string.
EndOfMessage		The available data area is always extended to the end of the data buffer.

Notes on determining the length of a data field 'numElements'

warning

imageType: ByteArray
The function for decoding data fields has not yet been verified. Use only after consultation!

Supported channel data types (target data types) 'dataType'

The dataType is always determined automatically from the imageType if it is not specified. A target format is set to avoid information loss with the smallest possible memory requirement.

Enum	Alias	Value range	Application
Boolean	bool	false/true	Status/control signals
Float		up to $3.40\cdot10^{38}$	Measurement data
Double		up to $1.80\cdot10^{308}$
Integer8	int8	$-2^{7}\dots2^{7}-1$
Integer16	int16	$-2^{15}\dots2^{15}-1$
Integer32	int32	$-2^{31}\dots2^{31}-1$
Integer64	int64	$-2^{63}\dots2^{63}-1$	Timestamp
UnsignedInteger8	uint8	$0\dots2^{8}-1$	Status codes
UnsignedInteger16	uint16	$0\dots2^{16}-1$
UnsignedInteger32	uint32	$0\dots2^{32}-1$
UnsignedInteger64	uint64	$0\dots2^{64}-1$
ByteArray
String			Text

Enforced data types:

If bitLength == 1 or imageType == bool, then dataType is automatically always bool.

For imageType == string or bytearray, the dataType is always string or bytearray.

The imageTypes time64 or timedate48 enforce the dataType with int64 in order to convert the timestamps correctly and completely in nanoseconds since 01/01/1970.

A specification of numElements > 1 enforces the dataType ByteArray, unless an imageType String is set.

Automatic data types:

Floating point formats float and double take precedence if the data source already provides a floating point or fixed point value or scale != 1.0, offset != 0.0 or a physicalUnit is specified. If bitLength <= 24, a float is sufficient for lossless conversion.

Integer formats are also selected via the bitLength and the presence of a sign (signed) in the imageType.

Manual data types and conversion:

warning

Manual selection of the dataType can lead to a loss of accuracy, resolution, or a significant restriction of the available value range and should therefore generally be avoided!

If the data source returns a floating point number according to the rules mentioned above, the conversion to a dataType integer is rounded up from .50 to the next larger integer value.

In any case, the value ranges of the selected integer types are taken into account. If the value from the image is outside the available range, the corresponding minimum or maximum that can be displayed is replaced and a message is output to the log file.

Data reduction filter

A data reduction filter is configured for each channel. In the default setting, this ensures that new data records are only written to the channel when the data content changes (OnChange). The reception timestamp of the data, on the other hand, is updated with each processed data record, enabling the (constant) data to be processed in other plugins up to the most recent point in time.

The absoluteTolerance option sets a tolerance band around the last written value. A new data record is only published if the difference to the last written value deviates by more than this tolerance band. This setting is helpful for highly noisy signals in order to significantly reduce the amount of data.

If the noFilter option is set to true, no data reduction filter is created. This setting is useful for fast data that must not be reduced, e.g., because vibration signals are to be analyzed in the frequency domain.

Module information

Information	Value
Authors	optiMEAS Measurement and Automation Systems GmbH
Since smartCORE	0.103
Module type	Fast Message Receiver, Producer
Dependencies	Fast Message Transmission Module (e.g., fmudp, canbus, smartmvb, ...)

The bufferSize specification determines how many samples in the smartCORE channel are reserved for use by other modules. It should be selected so that approximately 5 to 10 seconds can be buffered. Periodic data gaps in the OSF data indicate that the bufferSIze is too small. ↩ ↩²
If the "messageKey" is not configured, data from all messages is processed (this is useful for CAN RAW and debugging applications, for example). ↩
If "bitLength" is not configured, it is determined from the imageType or the length of the forwarded message. ↩
Mandatory only for imageType string ↩
Specifying scale != 1 or offset != 0 with a numeric imageType results in output with dataType float or double. ↩ ↩²
As a rule, the data type should be omitted, as it is then determined in an optimized manner from the imageType and the bitLength. For some combinations, the data type is mandatorily specified by the module. ↩
Specifying a cacheSize reduces CPU utilization for very fast data sources ( $> 100 Hz$ ), as new data sets are first stored in a cache and only published to other modules once the cache is completely full. The specification should be chosen so that publication occurs at a rate of $5$ to $20 Hz$ . ↩
Applies explicitly only to byteOrder bigEndian. For byteOrder littleEndian, the AscendingFirstBit scheme is counted. ↩ ↩²
byte 1 * 8 + rightShift 3 = 11 ↩
byte 4 * 8 + rightShift 1 = 33 ↩
byte 1 * 8 + rightShift 2 = 10 ↩

Description​

Interfaces & protocols used​

JSON configuration​

Example configuration (minimal)​

Example configuration (maximum)​

Global module parameters​

Configuration of a channel (JSON object)​

Supported address specifications 'addressSpec'​

Determining the anchor bit​

Boolean signals​

Supported byte order 'byteOrder'​

Use of a mapping string​

Supported source data types 'imageType'​

Notes on determining the length of a string 'stringHint' and 'numElements'​

Notes on determining the length of a data field 'numElements'​

Supported channel data types (target data types) 'dataType'​

Enforced data types:​

Automatic data types:​

Data reduction filter​

Module information​

Footnotes​

Description

Interfaces & protocols used

JSON configuration

Example configuration (minimal)

Example configuration (maximum)

Global module parameters

Configuration of a channel (JSON object)

Supported address specifications 'addressSpec'

Determining the anchor bit

Boolean signals

Supported byte order 'byteOrder'

Use of a mapping string

Supported source data types 'imageType'

Notes on determining the length of a string 'stringHint' and 'numElements'

Notes on determining the length of a data field 'numElements'

Supported channel data types (target data types) 'dataType'

Enforced data types:

Automatic data types:

Data reduction filter

Module information

Footnotes