Message Types

EchoStream requires that all messages that are processed in your Tenant's network be typed. The typing mechanism is soft (EchoStream does not enforce it) and is implemented via assigning Message Types to the Nodes in your processing network.

Message Types provide several capabilties in EchoStrem:

• A typing mechanism when connecting Nodes or choosing Functions.
• A audit capability that is specific to that Message Type, allowing for customized audit fields.
• Helper templates for custom creation of inline functions in Nodes.
• A set of requirements that will be included in all Internal Nodes and Functions that use the Message Type.

Each Message Type in EchoStream has the following attributes:

Auditor

EchoStream audits every message every time it arrives at a Node (and optionally, for some Nodes, when it is sent). For certain Message Types, enough is known about the structure of the message for EchoStream to augment the standard audit fields with information from the message itself. To do this, EchoStream uses the auditor provided by the Message Type.

For example, the echo.file Message Type is a JSON document with a specific set of fields. The auditor for echo.file is:

def auditor(*, message, **kwargs):

    import simplejson as json

    if not isinstance(message, dict):
        message = json.loads(message)

    return dict(
        content_length=message["contentLength"],
        content_type=message["contentType"],
        modification_datetime=message["modificationDateTime"],
        name=message["name"],
    )

This auditor will add the fields echo_file_content_length, echo_file_content_type, echo_file_modification_datetime and echo_file_name to the audit records for echo.file messages, thereby allowing you to perform more discrete analysis of echo.file messages as they are processed through your Tenant.

Templates

Some Nodes in EchoStream allow for customized processing of message that are received by those nodes. When custom code is selected as the processing mechanism, EchoStream will use the function templates provided by the Node's receive Message Type as a starting point for your function coding.

In most cases these templates are pretty basic, but they don't have to be. For example, the processor template provided by the echo.csv Message Type is:

def processor(*, context, message, source, **kwargs):

    from io import StringIO
    from csv import DictReader, DictWriter

    # Convert the csv into a list of dicts
    rows = DictReader(StringIO(message), dialect="excel")

    # If transforming csv to csv
    message_buffer = StringIO()
    # If you are changing the field struture, replace rows.fieldnames with your fieldnames
    writer = DictWriter(message_buffer, rows.fieldnames, dialect="excel")
    writer.writeheader()
    for row in rows:
        # TODO - Perform any transformations to the message here.
        # This can include transforming the message to something that
        # is no longer csv. Remember, you MUST return a string or None.
        # If None is returned, the message will be filtered.
        writer.writerow(row)

    return message_buffer.getvalue()

This template gives you the basics for reading and writing CSV messages in Python, allowing you to focus on the manipulation of the data into a different CSV format.

Requirements

Messages of certain Message Types may contain data that requires specific third-party Python packages to parse or process. EchoStream allows Message Types to specify these packages in the requirements field, which is a list of packages required by the Message Type.

For example:

• JSON-based Message Types will often require the simplejson Python package in order to gain advanced JSON functionality.
• YAML-based Message Types will often include the pyyaml Python package in order to ease the parsing and writing of YAML formats.
• XML-based Message Types will often include the xmltodict Python package in order to ease workign with XML data.

These requirements are pip requirements specifiers and must be accessible using a standard pip install ... call.

Requirements included in a Message Type will be included every time a Node or Function uses that Message Type.

README

Message Types include a README in Markdown format that explains the Message Type in a human-readable fashion

Sample Message

Message Types contain a sample message that is used in various locations that allow for validation of your custom processing code for that Message Type.

For example, the echo.json message type contains the following sampleMessage:

{
  "firstName": "John",
  "lastName": "Smith",
  "isAlive": true,
  "age": 27,
  "address": {
    "streetAddress": "21 2nd Street",
    "city": "New York",
    "state": "NY",
    "postalCode": "10021-3100"
  },
  "phoneNumbers": [
    {
      "type": "home",
      "number": "212 555-1234"
    },
    {
      "type": "office",
      "number": "646 555-4567"
    }
  ],
  "children": [],
  "spouse": null
}