Understanding streams

Proxima uses streams as a core component for streaming applications.

Streams have unique names and are built with generalized schema and distinct ordering so that they can be composed of events from different smart contracts and blockchains.

Naming Conventions

Streams within Proxima follow the template for naming. When using, searching, and producing streams, their names can be broken down as such:

<owner>.<scopes...>.<stream>.<version>

Example: proxima.ft.eth-main.new-tokens.1_1

• <owner> is proxima for our streams.

• <scopes...> is an arbitrary length (possibly empty) .-separated list of scope names which helps to group related streams together in an alphabetically ordered list.

• <version> helps to update the streams without breaking existing clients. It consists of two numbers — major and minor versions. If only the minor version is increased, the client can consume the new stream with the same code. If the major version is increased, it may require the client to change the way of stream events treatment.

• <owner>, <scopes> and <stream> consist of lowercase letters, digits and “-" symbol.

Concepts

Consuming streams with Proxima is simple, but it is helpful to understand some of the core components of streams and events. To fully understand the client design, you should first learn some facts about the event streams.

Handling reorgs

First, the streams are immutable (append-only). That means that after the event is produced to the stream, it can never be removed from it, even if no one has consumed it yet. But the sources of most of our streams are blockchains, which are not immutable. After publishing events coming from some blocks "X" and "Y," those blocks could be "forked", rewriting history with blocks "B", "C" and "D" instead. As we can't remove published events from the stream, we publish "undo" events ("-Y" and "-X" on the illustration) which logically cancel the effect of previous events. Each "undo" event corresponds to the last uncancelled event in the stream, has the same payload, and has undo property set to true.

Offset

Offset is some unique pointer to the place in event history. The offset actually points not to the event but to the state between the events. For example, an ordinary event followed by the "undo" event always leads you back to the same offset.

Every stream has an initial offset Offset.zero.

Height

We call a height of an offset the number of events directly preceding it, skipping any forks. Hence every regular event increases height by one, and every "undo" event decreases it by one.

Timestamp

Each event has a timestamp which is the time when the event actually happened in the world. For example, when the block was mined in a blockchain. It may differ from the time when that event got produced to the stream. The timestamp of an offset is the timestamp of the event preceding it.

Aside from epoch time in milliseconds, a timestamp may contain additional parts which help to strictly order the events with equal time. For example, the parts list may contain the index of the transaction inside a block and the index of emitted log inside that transaction.

Reading direction

When you try to read stream events starting from a given offset, most client methods will require specifying a direction of reading which can be either next or last. It controls the direction of consumption.

• The next mode starts reading events that happened after the given offset in the order they were produced. Note that as the offset points between the events, the first produced event in this mode may have a timestamp greater than the timestamp of the given offset.

• The last mode starts reading events backward from the given offset. In this mode, the first produced event will have a timestamp equal to the timestamp of the given offset.

Internals

Instead of having a single backend service, the client actually communicates to multiple services:

• Stream registry, which stores the streams' metadata and endpoints of StreamDBs for a given stream.

• StreamDB instances that store stream events.

To access them, there are two client classes:

• StreamRegistryClient which only accesses the stream registry and can fetch metadata.

• ProximaStreamClient which makes requests to both the stream registry and StreamDBs. It fetches a list of StreamDB endpoints from the registry and uses it to automatically connect to the correct instances and switch between them when necessary.