Update Twitch Users

[keeperxy]

Hey,

i’m searching for a way to update ~500k twitch users in my database.
Background: i’m programming a raffle bot and in each active channel i’m monitoring the users. thanks to the option for users to change their username i have to check regular if their username is the same or if they have changed it.
i have to do this cause im monitoring the “viewtime” of each user so you can consider this for your raffle.

the new api has a rate-limit and with this it took me about 7-8 hours for this users to check. the number will increase with every day. i’m hoping you have some ideas for me.

I’m using the following script:

total=$(mysql --login-path=local $DB_NAME -Bse "SELECT COUNT(*) FROM twitchs WHERE 
twitchs.id_twitch != 0;")
offset=0
echo "total: $total"
echo "DB: $DB_NAME"
while [ $total -gt '0' ]; do
    tmp=$(mktemp)
    header=$(mktemp)
    user=$(mysql --login-path=local $DB_NAME -Bse "select twitchs.twitchname from twitchs where twitchs.id_twitch != 0 limit 50 OFFSET $offset;" | sed ':a;N;$!ba;s/\n/\&login=/g')
    curl -s -H 'Client-ID: FOO' -H 'Authorization: Bearer BAR' -X GET "https://api.twitch.tv/helix/users?login=$user" -D $header -o $destination/twitchs_$offset
    #update users
    total=$(expr $total - 50)
    offset=$(expr $offset + 50)
    limit=$(awk '/Ratelimit-Remaining/ {print $2}' $header)
    if [ $limit = 0 ]; then
        time_limit=$(awk '/Ratelimit-Reset/ {print $2}' $header)
        cur=$(date +%s)
        wait=$(expr $time_limit - $cur)
        #echo "Wating $wait Seconds"
        sleep $wait
    fi
    rm -f $tmp $header
done

Regards
KeeperXY

[Dist]

From what I can tell, you’re only requesting 50 users at a time, why not 100 and instantly half the number of requests you would need?

Also, rather than requesting by username, it might help to request by ID. When requesting using login you wont get data for a user who has changed username, as that login would no longer be correct which would mean another lookup by ID to see what their new username is, so why not just do it all by id and then you’ll have their new username anyway.

Finally, you might want to consider request a rate limit increase. I had my request accepted (still waiting on it to actually happen though) as I also regularly check my user collection for username changes.

[keeperxy]

Oh, your right. I thought the limit was 50 user. I’ll change that.
I’ll also change to ID, right now it’s only a test script to see how long it takes.
And yeah, i’ll request for rate limit increase, thought maybe there was another way without it.

But thanks Dist