Streaming Protocol

Streaming is used to move large amounts of data (kilobytes and up) across a OpenLCB implementation.

This documents describes the streaming protocol(s) in OpenLCB.

Environment of Proposal

Requirements

Preferences

Design Points

Protocol

Stream Content Type Identification

Stream senders can, but don't have to, identify the content type (format, meaning, etc) of the data stream at the front. As part of their private protocol, they can use whatever structure they'd like for the data.

General content types can be defined using unique identifiers. The “Type Included” flag identifies that the first eight bytes of stream data are to be interpreted as stream content type information. These can be allocated via the same mechanisms as EventIDs. Well-known ones will be published by OpenLCB. These identifiers are recorded in a separate worksheet.

Buffer Size

The buffer size can be up to 216-1 bytes = 64KB-1. This is driven by the size of the initiate messages, which we want to keep in a single CAN frame. Although some transport mechanisms might be able to profitably use larger buffers, it seems unlikely that a layout control bus will need to have a higher message efficiency or lower latency than this size will allow. Should that turn out to be necessary in the future, an alternate form for the stream initialization messages can be defined for those large-message nodes. (It's unlikely they're running over CAN)

CAN Discussion

On a CAN segment, the data limit is 8 bytes. This requires use of as little control information as possible, so that as many of those eight bytes can be used for data as possible.

OpenLCB/S9.6 CAN frames can hold the source and destination aliases in the header. A dedicated “stream data transfer” format can also be coded entirely in the header. This then allows 8 bytes of data per transfer if the Source Stream ID and Destination Stream ID need not be carried. By specifying that only one stream can be transferred on a CAN link between any two nodes at a time, no Stream ID would be needed. This is our chosen solution. Since a node knows that it's originating the stream, this can be checked in advance for point-to-point connections. Gateways might have to reject a stream transfer in some cases, or use a virtual node ID.

If there's a need for more than two streams, another format could be defined that puts a source or destination stream ID in the data portion, transferring six bytes at a time. The sender could then use that form if a 2nd stream is required. That is left for a future option.

Implementation Notes

The "Stream Data Proceed" from the destination is clearance to send another buffer-size-worth of data. To achieve better performance, the destination can send it before receiving the entire buffer-size-worth of data, as soon as it has room to receive what's already been OK'd plus one more buffer size. For example, a destination with a 4kB buffer could reply with Max Buffer Size of 2K, followed by an immediate Stream Data Proceed, to do single overlap of the transfer.

Intermediate nodes need to be able to handle transfers, and therefore need permission to lower the Max Buffer Size on the outbound Stream Initiate Request message. The length in the returned Stream Initiate Reply can't be changed, as the destination need to know when to send clearance for another buffer's worth of data.

A CAN ↔ Ethernet bridge might receive several kilobytes of data from the Ethernet side at a time. It's then responsible for breaking that up into CAN frames and forwarding it. It can reduce the buffer size of transfers if need be to ensure there's a place to store the data while this is done.


Site hosted by

This is SVN $Revision: 1433 $