RFC 0017 - Introducing a new EventSub transport type: Conduit

[HariniB]

Summary

EventSub is a transport-neutral real time notification delivery service. EventSub was launched in 2020 with support for notifications via Webhooks. Last month, we launched support for EventSub notifications via WebSockets. Today, we’re proposing a new transport type called a Conduit.

Motivation

When we added WebSocket support to EventSub, we focused on supporting applications that run on end customer devices. We heard from backend application developers interested in receiving notifications over WebSockets on their backend servers.

Large backend applications often create and maintain millions of EventSub subscriptions across their user base. These applications need to load balance high throughput notifications across multiple server-side processes with resilience. Additionally, WebSocket subscriptions do not last beyond the lifetime of the connection. For backend applications with a large user base, short-lived subscriptions tied to the status of the WebSocket connection can result in sub-optimal user experiences when a connection is lost.

Proposed solution

Conduit transport type is a wrapper that separates your subscriptions from the underlying transport and load balances notifications across shards. A shard is a Webhook or WebSocket connection and a Conduit is a collection of shards. The Conduit transport type is for backend server applications and requires app access tokens. We will be introducing new EventSub APIs to manage the Conduit lifecycle.

Conduit Setup and Notification Flow

Conduit Setup

First, you create a conduit and specify the number of shards you need. If you want the conduit to deliver over WebSockets, you then create the desired number of WebSocket sessions. Then you associate your WebSocket sessions or Webhook callbacks with specific shards in the conduit. Here’s how that looks:

Associate your WebSocket session with a conduit within 10s of establishing the connection to avoid being disconnected.

Subscription Setup

You can add subscriptions to a conduit by specifying the conduit ID as the transport type when you create the subscription. EventSub routes notifications to your conduit shards. EventSub uses a hashing mechanism to determine the shard that notifications for a particular channel ID will be delivered on. Subscriptions are disabled when a conduit is deleted.

Subscriptions are associated with the new transport type via Conduit ID at creation.

Scaling

Developers can scale up and scale down based on user behavior. Use the EventSub Update Conduit API to scale up or scale down by changing the number of shards available to the Conduit. Here’s the flow for scaling up:

You can scale down by reducing the shard count on a conduit. For example, updating the shard count from 100 to 50 will disable shards with ids 50-99.

Failure Scenarios and Cleanup

EventSub can disable individual shards within a Conduit (e.g., after a websocket session disconnects). When a shard is disabled, we’ll send a notification via a new shard disabled subscription type. If your shard is disabled, you can create another session and update the conduit via API.

If all the shards associated with a Conduit are disabled, the Conduit will remain enabled. However, if no shard is reactivated after 72 hours (subject to change), EventSub will delete the Conduit. This allows developers to recover from full outages without needing to recreate every subscription associated with their system.

Authorization

Creating a conduit and associating subscriptions with it requires an app access token.

Limits

We will limit clients to 5 enabled conduits with a maximum of 20,000 shards per conduit (all numbers provided are subject to change). Subscriptions associated with conduits will use the same cost model EventSub has today.

FAQ

When will this be available?

We’re excited for beta participants in Q4 2023. Look for more details soon!

How long does a conduit persist?

Conduits persist while their associated shards are enabled.

Do subscriptions on conduits persist indefinitely?

No. if all shards are disabled for 72 hours (subject to change), EventSub will delete the conduit and revoke its subscriptions.

Can I specify the shard that EventSub should use to send notifications for a particular channel?

No. EventSub uses a consistent hashing algorithm to determine the shard. The number of shards associated with the conduit is an input into the hashing algorithm.

Can I create a Conduit with only one shard?

Yes, you can create a Conduit with one shard. Conduits will not have a minimum required shard count. We may in the future put in protections against hot shards.

Will notifications be distributed amongst the disabled shards? For example, If my shard count is 20,000 and I have 5 enabled shards will notifications be lost or will they be delivered on the enabled shards?

EventSub would likely drop notifications in this scenario. When EventSub sends a notification to a disabled shard, it retries once to send the notification on another shard. If the 2nd shard is also disabled, EventSub drops the notification. For best results, you should update your shard count as needed and avoid having disabled shards. We will create a new “shard disabled” subscription type so you’ll know when a shard is disabled and be able to react appropriately.

[prod]

When we use app access tokens to create subscriptions for websocket shards, am I correct to assume the webhook authorization model is used where the scopes granted to the client_id by end users is relevant (since there is no user token used in the subscription creation)?

Also, does this imply that a single websocket shard can now receive privileged data for multiple users (unlike normal websockets that are limited to a single user_id)?

Lastly, given app access tokens are used, does max_total_cost of 10 still apply to websocket shards?

[Marenthyu]

@HariniB

No, there will be a minimum number of shards required for a conduit.

I’m curious about this part - above, it is mentioned

@HariniB

We heard from backend application developers interested in receiving notifications over WebSockets on their backend servers.

but a “minimum” would discourage developers from using a conduit if all they need is a WebSocket connection in a firewalled server scenario, where webhooks are unavailable and User Access Tokens are not wanted, whilst being on low enough volume (or the existing infrastructure only) warrants no more than a single connection on a single thread.

Whichever arbitrary non-singular minimum is chosen, it would force developers to have a minimum scale active at any given time, effectively removing the benefit of being able to scale-down when no traffic is expected/no capacity is needed.

As active example, I can quote my own bot, which never exceeds over 50 Events per second, and the average is below 1 per second. I do not require multiple connections. I just want WebSocket on the Server side.

[BarryCarlyon]

@prod

Also, does this imply that a single websocket shard can now receive privileged data for multiple users (unlike normal websockets that are limited to a single user_id)?

That is implied by

@HariniB

EventSub uses a hashing mechanism to determine the shard that notifications for a particular channel ID will be delivered on. Subscriptions are disabled when a conduit is deleted.

This means the conduit as a whole gets multiple channel ID’s
But a single shard will get all the topics for a given user, but you don’t control the mapping of user(s) to shard(s).

[ginovva320]

@prod

When we use app access tokens to create subscriptions for websocket shards, am I correct to assume the webhook authorization model is used where the scopes granted to the client_id by end users is relevant (since there is no user token used in the subscription creation)?

That’s correct.

@prod

Also, does this imply that a single websocket shard can now receive privileged data for multiple users (unlike normal websockets that are limited to a single user_id)?

Yes, since app token authorization is used here, we would not restrict which users you are subscribing for (given you have their proper authorization).

@prod

Lastly, given app access tokens are used, does max_total_cost of 10 still apply to websocket shards?

No, that limit is specific to WebSocket subscriptions. Conduit subscriptions would follow the existing generic subscription limits.

[prod]

Appreciate the clarification @ginovva320 - have a couple more questions for ya:

After a conduit is created, how long do we have before all shards need to be registered? And what are the consequences of not reaching the minimum shard count in that amount of time?

[prod]

@HariniB

You can scale down by reducing the shard count on a conduit. For example, updating the shard count from 100 to 50 will disable shards with ids 50-99.

Will it not be possible to disable a specific shard by its id? For example, I may want to conduct maintenance to one server with outstanding shards, but these may not be the most recently registered shards (e.g., in 0-50 rather than 50-99)

 

@HariniB

We will create a new “shard disabled” subscription type so you’ll know when a shard is disabled and be able to react appropriately.

Is it guaranteed that this notification would avoid disabled shards, unlike regular subscription types?

[ginovva320]

@prod

After a conduit is created, how long do we have before all shards need to be registered? And what are the consequences of not reaching the minimum shard count in that amount of time?

I think there may be a slight misunderstanding here. When a conduit is created, the shard count provided must be at least the minimum shard count.

Following the creation, the only time limit enforced is that if 0 shards are enabled for 72 hours, the conduit would be deleted. In practice, this means that after creating a conduit, you must update 1 shard with a transport within 72 hours.

Aside from that, the consequence of not updating the number of shards specified in your shard count is that you risk notifications being lost. The flow we’d recommend is: create the conduit with n shards, update n shards with n transports, create subscriptions pointing to the conduit. This would ensure no notifications are dropped.

@prod

Will it not be possible to disable a specific shard by its id? For example, I may want to conduct maintenance to one server with outstanding shards, but these may not be the most recently registered shards (e.g., in 0-50 rather than 50-99)

Correct, arbitrary shards cannot be disabled. In the specific scenario you’ve given, we’d recommend updating the shards with a new transport before taking those servers down for maintenance. Multiple shards can use the same transport.

@prod

Is it guaranteed that this notification would avoid disabled shards, unlike regular subscription types?

No, there won’t be special dispatching logic for this subscription type. We’d recommend that when subscribing to this subscription type, you supply a different transport type so that the disabled notification isn’t dependent on your conduit.

[BarryCarlyon]

@ginovva320

Multiple shards can use the same transport.

Then this makes the minimum shard count irrelevant.

if you make the minumum number of shards (2) and point both shards to the same websocket, then the minimum count is circumvented.

[ginovva320]

True. We’d like to encourage high throughput consumers to spread load across multiple different shards, but perhaps that’s better done via documentation rather than enforcing a minimum.