I need a pagination/cursor example

[Joshua_Olufs]

Hi, in my app I need to search through all streams with a particular game. I can use the pagination->cursor field to cycle through all the results, however I can’t find any documented examples that show what is returned when there are no more results. On a couple barely related threads I’ve seen it mentioned that the cursor field is empty when the end of the data has been reached, but what does that mean? Is it an empty string or what?

Here’s my pseudocode:

cursor = “”;
do {
checkWaitOnRateLimit();
getNextBlock();
parseBlock();
cursor = newCursor;
}
while (cursor != “”);

My problem is I don’t know what to test for in the ‘while’ condition. I fear without knowing this my code may run endlessly. If someone could give a simple example of how to properly loop through result calls using the cursor I’d be super grateful.

[MrTurtle]

I think the easiest way to test it is to create an infinite loop based on your assumption and see if actually is an infinite loop. You can also loop until first returned hash is empty which won’t rely on the cursor value at al. (that way you’ll avoid while condition too).

What I did when looping through live streams is I check for a specific viewers_count number because with the current Rate Limit (which is 120 with a Bearer Token) I won’t be able to go through all 30k+ streams every minute. I theoretically could do it with 3 or even 4 separate tokens, but ¯_(ツ)_/¯

[BarryCarlyon]

@Joshua_Olufs

Hi, in my app I need to search through all streams with a particular game. I can use the pagination->cursor field to cycle through all the results, however I can’t find any documented examples that show what is returned when there are no more results. On a couple barely related threads I’ve seen it mentioned that the cursor field is empty when the end of the data has been reached, but what does that mean? Is it an empty string or what?

Basically, on the last page, you won’t get a cursor. That cursor could be blank or the “key” could be missing from the result completely. Normally there the cursor key is omitted. I test for “does json have a cursor and is that cursor not blank” for choosing if I should attempt to load the next cursored page or not.

Which accounts for if there was a fault and a weird or blanked cursor was returned instead of the key removed.

A cursor being present, means, there is another page to load.
No cursor means no more pages.

[Joshua_Olufs]

Thanks for your quick responses. I am now able to identify the end by checking for the existence of the ‘cursor’ field as well as its contents as noted by @BarryCarlyon. As for hitting the rate limit, it’s still a small problem but since this code is running from a 5 minute cron task, I can afford to spend a few mins waiting for my request count to reset. 30k results / 100 results per request / 120 requests per minute = 2.5 (round up to 3) minutes to search all requests.

My pseudocode now looks something like this:

cursor = “”;
do {
    checkWaitOnRateLimit();
    getNextBlock();
    parseBlock();
    if (newCursorExists())
        cursor = newCursor;
    else 
        break;
}
while (true);