Skip to main content

STATUS-PROTOCOLS

Abstract​

This specification describes the Status Application protocol stack. It focuses on elements and features in the protocol stack for all application-level functions:

  • functional scope (also broadcast audience)
  • content topic
  • ephemerality
  • end-to-end reliability layer
  • encryption layer
  • transport layer (Waku)

It also introduces strategies to restrict resource usage, distribute large messages, etc. Application-level functions are out of scope and specified separately. See:

Status protocol stack​

The keywords “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in 2119. See the simplified diagram of the Status application protocol stack:

Status application layer
End-to-end reliability layer
Encryption layer
Transport layer (Waku)

Status application layer​

Application level functions are defined in the application layer. Status currently defines functionality to support three main application features:

Each application-level function, regardless which feature set it supports, has the following properties:

  1. Functional scope
  2. Content topic
  3. Ephemerality

Functional Scope​

Each Status app-level message MUST define a functional scope. The functional scope MUST define the minimum scope of the audience that should participate in the app function the message is related to. In other words, it determines the minimum subset of Status app participants that should have access to messages related to that function.

Note that the functional scope is distinct from the number of participants that is addressed by a specific message. For example, a participant will address a 1:1 chat to only one other participant. However, since all users of the Status app MUST be able to participate in 1:1 chats, the functional scope of messages enabling 1:1 chats MUST be a global scope. Similarly, since private group chats can be set up between any subset of Status app users, the functional scope for messages related to private group chats MUST be global. As a counter-example, messages that originate within a community (and are addressed to members of that community) are only of interest to participants that are also members of that community. Such messages MUST have a community-wide functional scope. A third group of messages are addressed only to the participant that generated those messages itself. These self-addressed messages MUST have a local functional scope.

If we further make a distinction between "control" and "content" messages, we can distinguish five distinct functional scopes.

All Status messages MUST have one of these functional scopes:

Global scope​

  1. Global control: messages enabling the basic functioning of the app to control features that all app users should be able to participate in. Examples include Contact Requests, Community Invites, global Status Updates, Group Chat Invites, etc.
  2. Global content: messages carrying user-generated content for global functions. Examples include 1:1 chat messages, images shared over private group chats, etc.

Community scope​

  1. Community control: messages enabling the basic functioning of the app to control features only relevant to members of a specific community. Examples include Community Membership Updates, community Status Updates, etc.
  2. Community content: messages carrying user-generated content only for members of a specific community.

Local scope​

  1. Local: messages related to functions that are only relevant to a single user. Also known as self-addressed messages. Examples include messages used to exchange information between app installations, such as User Backup and Sync messages.

Note that the functional scope is a logical property of Status messages. It SHOULD however inform the underlying transport layer sharding and transport layer subscriptions. In general a Status client SHOULD subscribe to participate in:

  • all global functions,
  • (only) the community functions for communities of which it is a member, and
  • its own local functions.

Content topics​

Each Status app-level message MUST define a content topic that links messages in related app-level functions and sub-functions together. This MUST be based on the filter use cases for transport layer subscriptions and retrieving historical messages. A content topic SHOULD be identical across all messages that are always part of the same filter use case (or always form part of the same content-filtered query criteria). In other words, the number of content topics defined in the app SHOULD match the number of filter use cases. For the sake of illustration, consider the following common content topic and filter use cases:

  • if all messages belonging to the same 1:1 chat are always filtered together, they SHOULD use the same content topic (see 55/STATUS-1TO1-CHAT)
  • if all messages belonging to the same Community are always filtered together, they SHOULD use the same content topic (see 56/STATUS-COMMUNITIES).

The app-level content topic MUST be populated in the content_topic field in the encapsulating Waku message (see Waku messages).

Ephemerality​

Each Status app-level message MUST define its ephemerality. Ephemerality is a boolean value, set to true if a message is considered ephemeral. Ephemeral messages are messages emitted by the app that are transient in nature. They only have temporary "real-time" value and SHOULD NOT be stored and retrievable from historical message stores and sync caches. Similarly, ephemeral message delivery is best-effort in nature and SHOULD NOT be considered in message reliability mechanisms (see End-to-end reliability layer).

An example of ephemeral messages would be periodic status update messages, indicating a particular user's online status. Since only a user's current online status is of value, there is no need to store historical status update messages. Since status updates are periodic, there is no strong need for end-to-end reliability as subsequent updates are always to follow.

App-level messages that are considered ephemeral, MUST set the ephemeral field in the encapsulating Waku message to true (see Waku messages)

End-to-end reliability layer​

The end-to-end reliability layer contains the functions related to one of the two end-to-end reliability schemes defined for Status app messages:

  1. Minimum Viable protocol for Data Synchronisation, or MVDS (see STATUS-MVDS-USAGE)
  2. Scalable distributed log reliability (spec and a punchier name TBD, see the original forum post announcement)

Ephemeral messages SHOULD omit this layer. Non-ephemeral 1:1 chat messages SHOULD make use of MVDS to achieve reliable data synchronisation between the two parties involved in the communication. Non-ephemeral private group chat messages build on a set of 1:1 chat links and consequently SHOULD also make use of MVDS to achieve reliable data synchronisation between all parties involved in the communication. Non-ephemeral 1:1 and private group chat messages MAY make use of of scalable distributed log reliability in future. Since MVDS does not scale for large number of participants in the communication, non-ephemeral community messages MUST use scalable distributed log reliability as defined in this original forum post announcement. The app MUST use a single channel ID per community.

Encryption layer​

The encryption layer wraps the Status App and Reliability layers in an encrypted payload.

Waku transport layer​

The Waku transport layer contains the functions allowing Status protocols to use 10/WAKU2 infrastructure as transport.

Waku messages​

Each Status application message MUST be transformed to a 14/WAKU2-MESSAGE with the following structure:

syntax = "proto3";

message WakuMessage {
  bytes payload = 1;
  string content_topic = 2;
  optional uint32 version = 3;
  optional sint64 timestamp = 10;
  optional bytes meta = 11;
  optional bool ephemeral = 31;
}
  • payload MUST be set to the full encrypted payload received from the higher layers
  • version MUST be set to 1
  • ephemeral MUST be set to true if the app-level message is ephemeral
  • content_topic MUST be set to the app-level content topic
  • timestamp MUST be set to the current Unix epoch timestamp (in nanosecond precision)

Pubsub topics and sharding​

All Waku messages are published to pubsub topics as defined in 23/WAKU2-TOPICS. Since pubsub topics define a routing layer for messages, they can be used to shard traffic. The pubsub topic used for publishing a message depends on the app-level functional scope.

Self-addressed messages​

The application MUST define at least one distinct pubsub topic for self-addressed messages. The application MAY define a set of more than one pubsub topic for self-addressed messages to allow traffic sharding for scalability.

Global messages​

The application MUST define at least one distinct pubsub topic for global control messages and global content messages. The application MAY defined a set of more than one pubsub topic for global messages to allow traffic sharding for scalability. It is RECOMMENDED that separate pubsub topics be used for global control messages and global content messages.

Community messages​

The application SHOULD define at least one separate pubsub topic for each separate community's control and content messages. The application MAY define a set of more than one pubsub topic per community to allow traffic sharding for scalability. It is RECOMMENDED that separate pubsub topics be used for community control messages and community content messages.

Large messages​

The application MAY define separate pubsub topics for large messages. These pubsub topics for large messages MAY be distinct for each functional scope.

Resource usage​

The application SHOULD use a range of Waku protocols to interact with the Waku transport layer. The specific set of Waku protocols used depend on desired functionality and resource usage profile for the specific client. Resources can be restricted in terms of bandwidth and computing resources.

Waku protocols that are more appropriate for resource-restricted environments are often termed "light protocols". Waku protocols that consume more resources, but simultaneously contribute more to Waku infrastructure, are often termed "full protocols". The terms "full" and "light" is just a useful abstraction than a strict binary, though, and Status clients can operate along a continuum of resource usage profiles, each using the combination of "full" and "light" protocols most appropriate to match its environment and motivations.

To simplify interaction with the selection of "full" and "light" protocols, Status clients MUST define a "full mode" and "light mode" to allow users to select whether their client would prefer "full protocols" or "light protocols" by default. Status Desktop clients are assumed to have more resources available and SHOULD use full mode by default. Status Mobile clients are assumed to operate with more resource restrictions and SHOULD use light mode by default.

For the purposes of the rest of this document, clients in full mode will be referred to as "full clients" and clients in light mode will be referred to as "light clients".

Discovery​

The application MUST make use of at least one discovery method to discover and connect to Waku peers useful for the user functions specific to that instance of the application.

The specific Waku discovery protocol used for discovery depends on the use case and resource-availability of the client.

  1. EIP-1459: DNS-based discovery is useful for initial connection to bootstrap peers.
  2. 33/WAKU2-DISCV5 allows decentralized discovery of Waku peers.
  3. 34/WAKU2-PEER-EXCHANGE allows requesting peers from a service node and is appropriate for resource-restricted discovery.

All clients SHOULD use DNS-based discovery on startup to discover a set of bootstrap peers for initial connection.

Full clients SHOULD use 33/WAKU2-DISCV5 for continuous ambient peer discovery.

Light clients SHOULD use 34/WAKU2-PEER-EXCHANGE to discover a set of service peers used by that instance of the application.

Subscribing​

The application MUST subscribe to receive the traffic necessary for minimal app operation and to enable the user functions specific to that instance of the application.

The specific Waku protocol used for subscription depends on the resource-availability of the client:

  1. Filter client protocol, as specified in 12/WAKU2-FILTER, allows subscribing for traffic with content topic granularity and is appropriate for resource-restricted subscriptions.
  2. Relay protocol, as specified in 11/WAKU2-RELAY, allows subscribing to traffic only with pubsub topic granularity and therefore is more resource-intensive. Relay subscription also allows the application instance to contribute to the overall routing infrastructure, which adds to its overall higher resource usage but benefits the ecosystem.

Full clients SHOULD use relay protocol as preferred method to subscribe to pubsub topics matching the scopes:

  1. Global control
  2. Global content
  3. Community control, for each community of which the app user is a member
  4. Community content, for each community of which the app user is a member

Light clients SHOULD use filter protocol to subscribe only to the content topics relevant to the user.

Self-addressed messages​

Status clients (full or light) MUST NOT subscribe to topics for messages with self-addressed scopes. See Self-addressed messages.

Large messages​

Status clients (full or light) SHOULD NOT subscribe to topics set aside for large messages. See Large messages.

Publishing​

The application MUST publish user and app generated messages via the Waku transport layer. The specific Waku protocol used for publishing depends on the resource-availability of the client:

  1. Lightpush protocol, as specified in 19/WAKU2-LIGHTPUSH allows publishing to a pubsub topic via an intermediate "full node" and is more appropriate for resource-restricted publishing.
  2. Relay protocol, as specified in 11/WAKU2-RELAY, allows publishing directly into the relay routing network and is therefore more resource-intensive.

Full clients SHOULD use relay protocol to publish to pubsub topics matching the scopes:

  1. Global control
  2. Global content
  3. Community control, for each community of which the app user is a member
  4. Community content, for each community of which the app user is a member

Light clients SHOULD use lightpush protocol to publish control and content messages.

Self-addressed messages​

Status clients (full or light) MUST use lightpush protocol to publish self-addressed messages. See Self-addressed messages.

Large messages​

Status clients (full or light) SHOULD use lightpush protocols to publish to pubsub topics set aside for large messages. See Large messages.

Retrieving historical messages​

Status clients SHOULD use the store query protocol, as specified in WAKU2-STORE, to retrieve historical messages relevant to the client from store service nodes in the network.

Status clients SHOULD use content filtered queries with include_data set to true, to retrieve the full contents of historical messages that the client may have missed during offline periods, or to populate the local message database when the client starts up for the first time.

Store queries for reliability​

Status clients MAY use periodic content filtered queries with include_data set to false, to retrieve only the message hashes of past messages on content topics relevant to the client. This can be used to compare the hashes available in the local message database with the hashes in the query response in order to identify possible missing messages. Once the Status client has identified a set of missing message hashes it SHOULD use message hash lookup queries with include_data set to true to retrieve the full contents of the missing messages based on the hash.

Status clients MAY use presence queries to determine if one or more message hashes known to the client is present in the store service node. Clients MAY use this method to determine if a message that originated from the client has been successfully stored.

Self-addressed messages​

Status clients (full or light) SHOULD use store queries (rather than subscriptions) to retrieve self-addressed messages relevant to that client. See Self-addressed messages.

Large messages​

Status clients (full or light) SHOULD use store queries (rather than subscriptions) to retrieve large messages relevant to that client. See Large messages.

Providing services​

Status clients MAY provide service-side protocols to other clients.

Full clients SHOULD mount the filter service protocol (see 12/WAKU2-FILTER) and lightpush service protocol (see 19/WAKU2-LIGHTPUSH) in order to provide light subscription and publishing services to other clients for each pubsub topic to which they have a relay subscription.

Full clients SHOULD mount the peer exchange service protocol (see 34/WAKU2-PEER-EXCHANGE) to provide light discovery services to other clients.

Status clients MAY mount the store query protocol as service node (see WAKU2-STORE) to store historical messages and provide store services to other clients for each pubsub topic to which they have a relay subscription

Self-addressed messages​

Messages with a local functional scope (see Functional scope), also known as self-addressed messages, MUST be published to a distinct pubsub topic or a distinct set of pubsub topics used exclusively for messages with local scope (see Pubsub topics and sharding). Status clients (full or light) MUST use lightpush protocol to publish self-addressed messages (see Publishing). Status clients (full or light) MUST NOT subscribe to topics for messages with self-addressed scopes (see Subscribing). Status clients (full or light) SHOULD use store queries (rather than subscriptions) to retrieve self-addressed messages relevant to that client (see Retrieving historical messages).

Large messages​

The application MAY define separate pubsub topics for large messages. These pubsub topics for large messages MAY be distinct for each functional scope (see Pubsub topics and sharding). Status clients (full or light) SHOULD use lightpush protocols to publish to pubsub topics set aside for large messages (see Publishing). Status clients (full or light) SHOULD NOT subscribe to topics set aside for large messages (see Subscribing). Status clients (full or light) SHOULD use store queries (rather than subscriptions) to retrieve large messages relevant to that client (see Retrieving historical messages).

Chunking​

The Status application MAY use a chunking mechanism to break down large payloads into smaller segments for individual Waku transport. The definition of a large message is up to the application. However, the maximum size for a 14/WAKU2-MESSAGE payload is 150KB. Status application payloads that exceed this size MUST be chunked into smaller pieces and MUST be considered a "large message".

Copyright and related rights waived via CC0.

References​

  1. 55/STATUS-1TO1-CHAT
  2. 56/STATUS-COMMUNITIES
  3. 10/WAKU2
  4. 11/WAKU2-RELAY
  5. 12/WAKU2-FILTER
  6. 14/WAKU2-MESSAGE
  7. 23/WAKU2-TOPICS
  8. 19/WAKU2-LIGHTPUSH
  9. Scalable distributed log reliability
  10. STATUS-MVDS-USAGE
  11. WAKU2-STORE