So now that yet another API has been announced even though it seems like v5 has only just been worn in, I have a few concerns with paging in general. In the past, and even right now, almost every endpoint handles paging differently, in a few regards.
Requesting Each Page
Some endpoints such as get-channel-followers have a
cursor paging parameter; a nice, easy string to keep paging until it is empty. However, many other endpoints such as get-channel-videos don’t have a
cursor paging parameter to follow, instead it only has an
offset parameter. This behavior isn’t inherently bad. With the pages that return a model with a
total field, getting all pages/elements is trivial, but not all models have one, an example of this is the get-all-teams endpoint. In the scenario where there is no
cursor paging parameter to follow and no
total field in the model to calculate the number of pages to be returned, the only option left is to keep incrementing
offset until an empty page is returned. However, this leads into another major issue, the offset behavior in different endpoints.
Because endpoints are worked on by separate teams, the same parameters in different endpoints can behave extremely different. There seems to be some “maximum” that each endpoint enforces, some endpoints enforce this as an arbitrary cap and others as the amount of elements that can be returned. What’s even worse than this is how some endpoints behave when this max is exceeded. Some endpoints return a blank page, some return an error, some return the previous valid page requested, and some even overflow/wrap around to the beginning and return what appears to be a “valid” page. In situations where you have to poll the endpoint until an empty/null page is returned and there is no way to tell how many pages should be returned, this can lead to some very bad situations as you might expect.
What can be done to combat these issues is what I hope is obvious. standardize the way endpoints handle paging and paging parameters. Personally, I believe each endpoint should have have
offset parameters, and each page result should include a
_total field just like get-channel-followers. The behavior of the offset isn’t of much concern to me, just make it consistent.
Every endpoint handles paging differently, which causes a lot of unnecessary head aches. Every endpoint should be standardized to handle paging and parameter behaviors the same way to make everyone’s lives easier.
This conceptually should also extend to the entire API, and not just paging.