The API is broken and keeps sending incomplete data

API
1

I have posted this issue before… but it is still not fixed.

I am running the following code:

$client->get('https://api.twitch.tv/helix/streams', [
	'headers' => [
		'Authorization' => 'Bearer '.$token,
		'Client-ID' => $client_id,
	],
	'query' => [
		'game_id' => ["10872","10970","11406","16951","18902","23579","27098","29448","30639","313204","31568","458126","460167","461067","476449","488615","490500","490698","490709","494544","494995","496464","497385","497877","500626","502447","510578","510799","5218","5645","9041"],
		'language' => [],
		'type' => 'live',
		'first' => 100,
		'after' => '',
	],
]);

It fetches for a list of live streams playing the specific list of games. There are literally thousands of streams which fit these parameters; but it limits to the top 100. So what it should do, is return the first 100, then give me a “pagination cursor”, so I can get the next group of 100 by putting that cursor value into the after field. I loop through this 5 times, to get the top 500 streams that match these parameters.

Sometimes I will get proper results. However, more often than not, I will not get proper results. Instead of getting the first 100… I will instead get around 40-60 results, and it will return a VERY SPECIFIC cursor value:

eyJiIjp7IkN1cnNvciI6ImV5SnpJam95TVRjeExqazBNak0xTVRneU5EVTVNRE1zSW1RaU9tWmhiSE5sTENKMElqcDBjblZsZlE9PSJ9LCJhIjp7IkN1cnNvciI6IiJ9fQ

This pagination cursor specifically means there are no more pages. But as I said, there should be over 500 streams right now that fit my parameters. I can keep on re-running the script, and eventually I will get proper results… but I shouldn’t have to do this.

2

Best advice is to do one game_id per request instead

it’s more reliable to me at least. Though generally I never specify more than one game_id, I prefer to parallelize my threads and run one thread per game ID

You are like the third person recently (do LOTS of game_id’s and have a issue fetching streams) to try to do this and had the same issue.

You may want to open a GitHub on the matter

3

That’s a quick way to get rate limited. Especially as the last of games expands.

4

Then keep within the rate limit.

This tool I wrote

https://barrycarlyon.github.io/twitch_misc/examples/browse_categories/

and converted for Server use, works quite well, but generally, I don’t wander north of 20 games anyway.

You can always set a max number of threads to run and employ logic to stay within the rate limiting. And get the data you need.

Since as you finish getting pages for game x, that’ll free up more threads for other games.

Additionally depending on why you are doing this, you probably can stop after 50 pages as the long tail is less statistically significant than the short tail.

5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.