The problem of acquiring only 500 cases

[hiros00]

Hello.

I want to retrieve video data for a university research project, but I can only retrieve 500 videos.

The documentation says “When searching by game_id,the response contains a maximum of 500 videos that show this content.”
Is it impossible to retrieve more than 500?

The language used is python.

Please help me to solve this problem.

def get_all_streams(cursor, counter=7):
    while counter != 0:
        url = 'https://api.twitch.tv/helix/videos?game_id=516575&first=100&type=archive&sort=views' + (f'&after={cursor}' if cursor else '')
        headers = {
            'Client-ID': client_id,
            'Authorization': f'Bearer {access_token}',
            'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8'
        }

        response_json = requests.get(url, headers=headers).json()

        if counter == 1:
            return data
        
        data.extend(response_json['data'])
        print(len(data), cursor)

        print("--------------")
        if 'cursor' in response_json['pagination']:
            time.sleep(1)
            return get_all_streams(response_json['pagination']['cursor'], counter - 1)
        else:
            print(counter)
            return data

get_all_streams(None)

df = pd.DataFrame(data[1:], columns=data[0])
df.to_csv("out.csv", index=True)

[BarryCarlyon]

@hiros00

Is it impossible to retrieve more than 500?

You answered your own question

@hiros00

The documentation says “When searching by game_id,the response contains a maximum of 500 videos that show this content.”

With this API and using a game_id you cannot get more than 500 videos

[hiros00]

Thank you for your reply.
That’s certainly true.

If I use user_id, less than 100 will be output, even though the user’s channel has more than 100 videos.

Do you know anything about this issue?

[BarryCarlyon]

@hiros00

If I use user_id, less than 100 will be output, even though the user’s channel has more than 100 videos.

If you are looking at the users page you might be thinking clips are videos, and clips show on the clips API

Or you are comparing highlights and archives and uploads on the page but only asking for archives via the API

Or some other combination

[hiros00]

Thank you for your response. I really appreciate it.