Hello, I have a question about pagination cursors for APIs such as streams.
For example:
Yesterday I made 100 requests to get the 100th cursor of a page containing 100 streams. I stored those cursors in a file. Channels had around 20 views on 100th page (entries from 10000-10100).
Today I tried to get the 100th page again with the cursor that I stored yesterday and I seem to have received the 100th page successfully.
Making a request takes around 0.2s, so “caching” pagination cursors like this saves a lot of time (kinda O(1)).
Is this a good way to minimize API calls for things that rely on pagination? Or do cursors expire?
Cursors are not intended to be stored. So if you go through 100 pages, save the cursor, and then at a later point try the cursor again it may work but there’s no guarantee it’ll correctly be the page you think and may simply not work at all as you’re not supposed to be using cursors like that.
but if, for example, a request takes longer to be received by twitch api server, that cursor still works. or are you saying that even the cursor, that i receive and instantly use for the next request could technically be “invalid” or at least the response that i get with that cursor could give me not “real” data?
There’s no more specific guidance about Cursors, other than that they’re intended to be used when you receive them and not stored and used at a later time.
Here are a couple examples, If you’re trying to periodically go through an endpoint with a lot of pages, eg Get Streams every 5 minutes, you’d start from the first page and use the cursor it gives you on each request, then 5 minutes later start from the beginning and do it again. You shouldn’t save the cursors and reuse them every 5 minutes. (Get Streams also gets progressively inaccurate as you page deeper, so in this example using stale cursors could throw things off even more).
If on the other hand you have an app that lets users view pages of results, and select next/previous page in their own time, then if seconds pass by before they move on that would not be an issue as the API is cached so the cursor is likely still fine. If minutes pass before the user selects to go to the next page, there’s the potential for inconsistent results as the cursor is stale.
will experiment around with this a bit more. maybe will try adding/removing cursors from local cache based on some arbitrary time period to see how that goes, and if something goes wrong, fallback to full “from page 1” procedure.