RFC 0015 - Mock Twitch API via CLI

Summary

The Twitch CLI has been designed with the mission that it should empower developers to easily test and simulate the Twitch APIs, whether that’s EventSub, WebSub, or by directly calling the Twitch API via command line.

We’ve seen the community take to the CLI better than we could’ve imagined, and we’re looking to add a full mock API implementation into the CLI.

Motivation

We’ve heard consistently that developers would like to test against the Twitch API without requiring affiliate/partner accounts. The most recent examples coming from the Channel Points hackathon, as that API requires the use of an affiliate account to test the creation and redemption of events.

Consistency with production is the most important factor when using it; developing/testing against a mock API that doesn’t match expected production results is a frustrating experience as a developer.

With that in mind, we believe the CLI is the best place for this mock implementation for a few reasons:

  1. It can be run offline/disconnected from the broader web, allowing users to develop on limited internet connections or used in CI/CD tooling
  2. It allows us to keep it simple and give an individual a sandbox to operate in without requiring shared resources

Proposed Solution

The CLI will be able to fully mock all Twitch API endpoints, including subscriptions and channel points.

As stated, we will fully implement all current Twitch API endpoints, along with matched production constraints (e.g. must provide a broadcaster_id for some endpoints).

These endpoints will be hosted locally via a local webserver and follow identical paths to allow for easy replacement within client libraries to enable easier development against. As this is a reimplementation of a currently-available API, there will be no major changes other than the ones covered below.

The CLI will allow developers to create fake users, subscriptions, and other principles for use with the APIs.

To allow developers to effectively use the mock endpoints, in addition to the public endpoints offered, we will also allow developers to create “principles”, or important constructs for use within the API. For example, you could recreate a user locally as a way to develop with specific users in mind. Additionally, we’ll provide methods to create faked subscriptions (and other monetary events) to be able to use in fetching later.

Detailed Changes

Authorization

For the initial implementation of this feature, authorization will fall out of scope for a local-only mock API. What this means is that the CLI will check for the existence of both the Client ID and Bearer token, but not confirm validity, scopes, or matching client IDs.

Mock API Implementation

Beyond authorization changes, the API will map to the Twitch API as close as possible without requiring an internet connection. Each endpoint will follow the same naming conventions, including parameters.

In the future, we will automate the drift between the CLI and production API to avoid discrepancies.

CRUD APIs for Principles

As noted above, one of the other portions of the mock API featureset will include the ability to manually create specific items. This will look like:

POST https://localhost:3000/api/user

-- Request Body --

{
     “user_id”: “1234”,
     “user_login”:”concreteentree”,
     “user_name”:”ConcreteEntree”,
}
-- Response Code --
204

For each of the endpoints, there will be supplied documentation within the CLI docs, as well as helpful error messaging to provide information on how to fix the issue.

WIth these APIs, they’ll be written to the existing SQLite database for use with further principles/calls. For example, if you were to call Get Subscriptions for the above user_id, it would return an empty array until you create a subscription for the example user.

FAQ

Will the mock API be a true 1:1 implementation of the Twitch API?

Not exactly, but close. The major caveat will be around Authorization, discussed above.

Why? Simply put, this would require one of two things; the first would be an internet connection to check against the production OAuth endpoints. Given this is against the spirit of the functionality, it doesn’t feel like a good fit. The second would be a reimplementation of the OAuth flows (or some mock implementation thereof), which could cause feature bloat and confusion on how OAuth works in production.

As a result, this is simply allowing you to mock out the responses from the API and test that your code is working properly so long as you get data.

5 Likes

Would new feature support be release in the CLI for developers to test against prior to a new feature going live on the Twitch?

1 Like

Very happy to see this.

However: given authentication is one of the most important pieces of the interaction with twitch, one of those that gets a lot of questions, and the thing that you need to make sure you get right, I am of the opinion that leaving it out is a large disservice.

I understand if its not in immediately, but I am in favor of this bloat if it means that I can mock the full auth process without having to hit twitch’s production servers (as I do now).

Mind, having the ability to disable it (just checking, but not validating tokens etc) would be ideal if that is not the component being worked on

1 Like

Would new feature support be release in the CLI for developers to test against prior to a new feature going live on the Twitch?

Potentially! Given we have control over the implementation in the CLI, we can also test out new features earlier, but that’s TBD.

Very happy to see this.

However: given authentication is one of the most important pieces of the interaction with twitch, one of those that gets a lot of questions, and the thing that you need to make sure you get right, I am of the opinion that leaving it out is a large disservice.

I understand if its not in immediately, but I am in favor of this bloat if it means that I can mock the full auth process without having to hit twitch’s production servers (as I do now).

Mind, having the ability to disable it (just checking, but not validating tokens etc) would be ideal if that is not the component being worked on

Fully agreed with this. For the first iteration, we’re going to offer two options. The first is an online experience whereby we use the production service to validate tokens to avoid fragmentation of mock auth vs. real auth, and the second is the auth-less option.

This way we can allow you to test end-to-end.

If you have specific feedback on what you’d like to see on a mock auth solution, let us know; always happy to look into it. In general, our concern is around user confusion more than anything, especially since it’d be overly simplified vs. the production flows. Just something we’re considering when working through this.

1 Like

Ideally I would like to see something that looks identical, or at least very similar to the prod site+flow (mind, at localhost:3001, etc.), except instead of entering a username and password on the login page, you just enter the username. Assuming the user exists, it proceeds mirroring the prod site+flow again.

The biggest confusion that I could see here would be the generation of client IDs/Secrets, (having seen several people ask how to generate one via the api), and the redirect URLs. Assuming all endpoints used a /mock/ prefix then I see no reason that creation them via the mock api would add too much confusion.

1 Like

I love this idea, and am looking forward to trying it out and incorporating it into a bunch of tests without having to ping the production server.

I think my only question is, will the port that the mock API uses be configurable? I know on test systems especially, other ports can get used up by similar mocked servers and APIs, so having it be configurable would be a nice option to avoid any conflicts.

1 Like

I think my only question is, will the port that the mock API uses be configurable?

That’s the plan! It’s going to be important for this, but most of the documentation will assume the default port; it’s a good callout regardless.

1 Like

I realize I should be explicit in mentioning/asking this: Will all endpoints to create specific items use a common path prefix such as /mock/?
The example appears to use the /api/ path prefix, but I fear that may confuse people using it for the first time.

Will an official docker container be published that launches the server?
This will be useful for ci/cd pipelines to not have to build their own container, but its also not the largest problem.

Will v5 be part of the “all Twitch API endpoints” that are getting mocked?
While deprecated, there are a number of things that can still only be done via V5.

I realize I should be explicit in mentioning/asking this: Will all endpoints to create specific items use a common path prefix such as /mock/ ?
The example appears to use the /api/ path prefix, but I fear that may confuse people using it for the first time.

We’ll likely use /mock/, as that’s what the community is suggesting here; definitely hear you loud and clear.

Will an official docker container be published that launches the server?
This will be useful for ci/cd pipelines to not have to build their own container, but its also not the largest problem.

Potentially, as it’ll be through the CLI and could be quickly exported to Docker.

Will v5 be part of the “all Twitch API endpoints” that are getting mocked?
While deprecated, there are a number of things that can still only be done via V5.

Likely not.

1 Like

We learned about this proposal today from @jbulava during the Twitch Dev Stream. We are fairly new to the Twitch development community, but already developed some open-source Twitch projects which put us in a position to comment on this proposal. At least so we think … :slight_smile:

tldr

  • Mocks are hard to build and maintain.
  • If you end up building a mock, use existing solutions like Postman.
  • Developers ask for a mock, but they might be better served by Twitch maintained & tested client libraries.

Keep reading if you’re interested in learning more about our thinking.


Developers would like to test against the Twitch API without requiring affiliate/partner accounts.

Oh yes. We are still 8 followers away from affiliate level, which prevents us from fully testing our Golang PubSub library. Unfortunately the PubSub docs appear to be outdated, which doesn’t help either. Dkamps18 created a ticket for it.


[The mock] can be run offline/disconnected from the broader web, allowing users to develop on limited internet connections or used in CI/CD tooling.

Limited internet connections can be found on trains in Germany (German expat here). We’d recommend enjoying the train ride instead of trying to work though if you ever find yourself on a German train. Look at the :european_castle: at 1:35!! We can’t speak for other - maybe less developed - places, but if every Twitch developer is also a Twitch streamer or broadcaster, slow internet is going to be a big problem in the first place. Does the mock really need to address this problem?

Our bet is that most Twitch developers use readily available CI tools like Github Actions. Github Actions allow egress connections. We’d love to find out, but even game studios like Blizzard will most likely not block egress traffic in their internal CI tooling. We could see a world where Coinbase or other financial or healthcare institutions firewall away their CI suite, but are they really a user-story for this mock?

The point is, from our perspective “not having a (slow) internet connection” doesn’t appear to be a good product requirement.


[The mock] allows us to keep it simple and give an individual a sandbox to operate in without requiring shared resources.

Mocks are hard to build and even harder to maintain, especially if you’re also planning on implementing pseudo-logic to create “fake users, subscriptions, …”. You will be implementing a mini version of your production API! Even if you start with a small feature set today, think about the future feature requests flying in, asking for more pseudo-logic to test against. Based on your comments, you are already anticipating a drift between the mock and the production API. If the mock returns responses inconsistent from the production API, developers will lose trust in the mock and instantaneously resort back to the production API for testing. That’s what we would do.


So, if you really, really have to build a mock, we’d recommend looking into tools that allow mocking of APIs. Postman is a good example. People have mixed feelings about Postman and suggest alternatives. VCR is something we used in the past. There are many ports to other languages, including Golang. Heck, you could even roll your own reverse proxy to mock things out. Why reverse proxy? That’s how most of these solutions work. They record a real production API response and let you replay it for testing purposes.

With all that said, we think the root problem is something else. After a sharp analysis on our end (jk, we just looked at what libraries exist today - work in progress), we found many client libraries across all programming languages implement the same thing. This takes away from our developer experience, because we always have to make an educated guess which library actually works. The fast way is to trust the GitHub stargazer count. The only correct and slow way is to review the code. Reviewing code takes practice and experience with the Twitch APIs, something a new developer lacks. A dilemma.

We’d love to make a counter-proposal:

  1. Twitch should build and maintain official client libraries for the most popular programming languages. Twitch engineers can test against the real production API if necessary, also having a deep understanding of how things work behind the scenes. This should make the mock obsolete.

  2. Twitch should offer a sandbox environment for API calls [1] that have (expensive) side effects, i.e. for subscriptions, channel points, and so on. It would be accessible to all developers, regardless of their status (affiliate/partner). The sandbox environment could also be described as a “Twitch playground” where a developer can trigger certain actions from within the developer console. Think “trigger test webhook with this payload”. In fact we love the playground idea so much, we thought about building it ourselves.

  3. Twitch should provide up to date documentation. It’s so important for newbies and should include guides, code examples and an API reference. The reference is necessary, because after all it is impossible to have client libraries for all programming languages, see [2] and [3].

To summarize: We’d love to see official client libraries instead of a mock. A sandbox environment would be helpful for integration testing. Good docs with tutorials and guides would be great.

We hope our feedback was helpful and we very much appreciate all of the hard work that goes into building Twitch and its community.

Tracy and Matt


PS: A mock for Chatbots/ IRC exists here: GitHub - fdgt-apis/api: `fdgt` is a mock API for the Twitch. Via `fdgt`, you can simulate events without having to spend a penny!

[1] Think Stripe Sandbox vs Production Mode, which can be toggled in their dashboard.

[2] There are ways to auto-generate client libraries from “templates”. Swagger might be a good tool to research. GRPC will take you to the next level, but can’t be retrofitted easily. Google has great success auto-generating their client libs. Have a look at their GitHub repo for example. The auto-generated libs provide low-level API access, handling API requests/responses in a generic way. Google then offers “hand-written” wrappers around the auto-generated libs for ease-of-use. [We just realized we should’ve used AWS examples :slight_smile: ]

[3] Google offers Emulators. It’s not a mock though. Emulators tend to be helpful for integration testing and local development.

3 Likes

I like the detail you go to in your post, but I’d like to add my own points to it:

  1. I agree that official libraries would be amazing, but in my past experience, I have been a lot more opposed to using an “SDK” of sorts a company publishes instead of rolling my own code as it always does things in ways that are not always quite what i want or feel like they “bloat” my needs - I don’t always want to run a local webserver or connect to pubsub when all i need is to exchange an id to a name.
  2. While a Sandbox environment would definitely be amazing, i personally would prefer the mock approach i can run and setup myself. It feels like i am more in control of it that way. Also I do think that a sandbox environment would be a more complicated solution to build from Twitch’s perspective, as it costs them more resources compared to letting everyone run their own one.
  3. As for “trigger test webhook with this payload” - that’s exactly what the CLI already does - at least for eventsub. I only recently touched eventsub for the first time myself and the CLI was a great way to verify I did the correct thing. Discord had a similar thing, though it was integrated into their oauth client configuration (for slash commands, when you set your callback URL in the web console, they send you a verification payload, allowing you to verify you did things correctly.)
  4. I agree that the docs must be kept up-to-date.
1 Like

All of this requires you to have gotten a valid API response to “record”. Which you can’t get, unless you use the ones in the example for the endpoints you can’t call as a non affiliate+

And a “rest client” should be just a “rest client”. I’ve spent more time in the IGDB discord helping people getting stuck trying to transition off Postman to production than I care to count. So adding a recommened tool that will just confuse developers doesn’t help. Of course you/other developers are free to do what they want to test API’s or self mock them.

  • Forcing Postman, adds in something that people don’t understand and get stuck when transitioning

Using Twitch CLI as a mock means

  • You just swap the URLs you are calling/Authing via and your test code works

And to push this further, people get stuck using the Developer Rig for Extension Development, and in most cases the problem is they have never done developing before and trying to jump in the deep end.

Yeah if all libaries for all languages are implementing the same API then they’ll all do the same thing, and just be “opinionated” on what the output looks like and how they handle key generation/handling. And be “opinionated” on how some parts of the API are exposed.

Personally I don’t use a Twitch Lib, why do I need a Twitch Lib to make cURL requests, I just throw it at php curl or nodeJS got and process the response?

Teach a developer how to use cURL and they can use any API without needing a library.

Shouldn’t be teaching people how to “use a library” - case in point I dispise how the Twitch Chat docs immediately go “go to Token Gen site to get a token then use tmi.js to get a bot going”

Cool you just taught the developer nothing of value other than how to use get a token from someone else that dies and can’t be auto remade, and how to use a library.

This is like teaching people to use a calculator and not how to do math on paper. (Clumsy example but should prove the point).

Should Twitch create it’s own libraries? Perhaps
It’s it useful to most developers? I know I wouldn’t use it.

And comes back to the “teach a developer to fish” example

According to you, Twitch might have someone who has expereince making them so are they hard to maintain if you know what you are doing?

Postman can die in a fire (respectfully)

No I don’t use libraries, I just call the API direct. (Granted via a library that implements cURL for me rather than syscalls, but with that curl library I can call any api, I don’t have to use a Twitter Library and a Discord Library and a Twitch library if I can talk to all three with just cURL, which saves memory, but too many devs library first when if they knew how to fish… they wouldn’t need a library and/or have to deal with the libraries interpreted errors)

I managed to build my implementation using the docs fine, sure the docs need to be clearer but theres enough there for a developer to get started, just need to wrap your head round it.

And some of the developers we help can’t do that since they are trying to jump in the deep end with not prior knowledge (see also “teach a developer to fish”)

What about planes?

On Planes and trains I generally assign that time to building things.

I’m assuming you mean “streaming” here, why would I try to stream from a train or plane?

Yes.

I disagree. See next point

Yes a sandbox is something that has been requested for years.
But a sandbox doesn’t work when you have no/poor network for whatever reason.
(Think hack days, hack days are notorious for “net got overloaded coz too many geeks”)

Yeah that is a great example of a sandbox done right.

Given Stripe devs were basically PayPal devs. It’s based on some of the work PayPal did with their sandbox, which is fudgy but workable

TLDR:

This kinda got jumbled in order. But it is what it is.

I’d rather Twitch offered all possible Tools to give the developer the most flexibility to the developer.

  • Add a mocking tool for those that want to mock
  • Add a sandbox so that people can test in there if they want
  • In house libaries, I’m in two minds here since “teach a developer to fish” and they can use any API. Teach a developer a library, and they learn nothing they can transfer.
  • And finally, yes, Documentation needs to be up to date as possible on the things third parties are allowed to use.

Am I more likely to use a Mock or a sandbox? Depends am I online or not.
Would I use a twitch made (or developer made library) no. (Aside from a cURL lib)

1 Like

Appreciate the details here!

Agree on many points here, but wanted to provide a broader image as well to explain why we believe mocks are the right thing (for now).

Oh yes. We are still 8 followers away from affiliate level, which prevents us from fully testing our Golang PubSub library. Unfortunately the PubSub docs appear to be outdated, which doesn’t help either. Dkamps18 created a ticket for it.

Sidenote- if you have specific feedback, feel free to message me on Discord as well; Same username in the Discord server. :slight_smile:

Our bet is that most Twitch developers use readily available CI tools like Github Actions. Github Actions allow egress connections. We’d love to find out, but even game studios like Blizzard will most likely not block egress traffic in their internal CI tooling. We could see a world where Coinbase or other financial or healthcare institutions firewall away their CI suite, but are they really a user-story for this mock?

The point is, from our perspective “not having a (slow) internet connection” doesn’t appear to be a good product requirement.

I can agree with this point to a degree, and it doesn’t really immediately solve the problem, however Twitch is not just smaller developers; we have working relationships with many large multinationals that integrate with our APIs (think like Riot) and these customers are more where avoiding an Internet connection is a plus.

It’s not the entire reason for putting it in the CLI, however- beyond the portability of the data that’s been noted by Barry and Maren below, there’s also the aspect of community engagement that’s not currently available on the docs or otherwise. Meaning if there is drift, for whatever reason, it can be fixed very quickly by myself or others within the community.

There are other reaosns that I’m not sure I can really dive into about this as well, but regardless- it’s an approach that allows us to test the idea quickly. And who knows? Maybe it does get absorbed into an official Sandbox down the road.

If the mock returns responses inconsistent from the production API, developers will lose trust in the mock and instantaneously resort back to the production API for testing. That’s what we would do.

Agree! This is the piece we’re working on figuring out the right process for since this sort of client-side mock server is very atypical. It’s a priority for us to address and address in the right way since it can erode trust- but there’s also the side that means we could have the power to test new functionality prior to going live as well, in addition to having blessed schemas for the API results due to Golang’s strong typing.

To summarize: We’d love to see official client libraries instead of a mock. A sandbox environment would be helpful for integration testing. Good docs with tutorials and guides would be great.

We hope our feedback was helpful and we very much appreciate all of the hard work that goes into building Twitch and its community.

I appreciate the candid feedback here; it’s definitely much appreciated and something we’ll read into. Needless to say, most of this is not explicitly new, but regardless refreshing to see the thoroughness you’ve put into it.

1 Like

As the CLI is open source!

Hopefully it’ll stay open source (unlike the dev rig before it)