Bulk Data Storage

EchoStream messages are limited to 256KB of data. While this is a fairly large message in most messaging protocols, there are notable examples. For example, what if you want to pass a large CSV file from a mainframe, or a medical imaging object (DICOM) from an x-ray scanner? It is likely that both of those messages could easily exceed 256KB.

EchoStream provides Bulk Data Storage to accomodate for such large messages. With Bulk Data Storage, you provide EchoStream with the binary object that you want to pass through your processing network and EchoStream provides you with a URL that allows for secure access to that binary data as many times as you wish. You may think of this URL as a secure ticket that grants access to your information.

All data stored in Bulk Data Storage is encrypted in flight and at rest, is obfuscated through auto-generated names, and is compressed using either the gzip or the deflate algorithm. Futhermore, each Tenant receives it's own Bulk Data Storage location that is not shared with any other Tenant.

Size limitations on Bulk Data Storage

Each call to the EchoStream Bulk Data Storage is limited to 5GB of data. However, since you can make as many requests of the Bulk Data Storage system as you wish per message there is really no practical limit to the size of a message that you can process through EchoStream.

Bulk Data Storage lifecycle

Every object that you store in Bulk Data Storage will be retained for 7 days and then automatically removed. You may not remove Bulk Data Storage explicitly; given that messages can fan out to an unlimited number of Nodes, knowing that a particular Bulk Data Storage object is no longer required is effectively impossible.

Best Practices

When dealing with a message that you want to store in Bulk Data Storage:

• Do not place data about the message that you will require to route the message or to perform normal processing into Bulk Data Storage, as retrieving these large objects while routing or processing could dramatically slow down both.
• When receiving a message that contains a Bulk Data Storage URL (i.e. - ticket) in a terminal Node, be sure to GET the data immediately. Remember, Bulk Data Storage objects will disappear 7 days after their creation.
• When accessing an object in Bulk Data Storage, use an HTTP library that automatically decompresses the object based on the Content-Encoding header. Remember, all Bulk Data is compressed with either gzip or deflate.