PSA: Chat servers are going to migrate to AWS EC2 servers

will · 2016-03-23T02:24:14.345Z

We were just discussing this today! I’ll check it out.

will · 2016-03-23T02:25:50.272Z

We’re only moving PS4 channels back, because we have a fix for them which will be deployed tomorrow (pointing irc.twitch.tv at irc.chat.twitch.tv). PS4 already uses the correct port for this to work (6667).

By the end of the week there will only be one cluster (the new servers), so this bot will need to support the new environment. Send me an email at will@twitch.tv (or have the bot developer do so), and I can help get it working.

superstringman · 2016-03-23T02:25:53.074Z

It seems to be with the api endpoints, for followers and tmi for chatters. We have some issues on our end that i need to figure out , but what seems to happen is we get a bunch of timeouts on requests, which is causing our network queues to fill up and bring us down. I’m still trying to figure it out, so not a lot of details yet.

daytona_675 · 2016-03-23T09:28:34.939Z

Hi Will,

Today my channel was migrated to the new servers around 6pm PST. Updating the hostname to irc.chat.twitch.tv made normal chat functions work properly for my bot (im a dev).

My issue is that whispers stopped working. They were working fine before. To be clear, I’m only sending whispers with a bot, not receiving them.

I know the PSA says group chat is not affected, but its not working anymore

I’m getting my server IP from http://tmi.twitch.tv/servers?cluster=group (Tried all of them)

I’ve also tried using IPs from http://chatdepot.twitch.tv/room_memberships?oauth_token= (Tried a few of them)

The connection to the group server seems normal. My CAP REQ is ACKed as expected. I don’t seem to get any PINGs and my whispers do not get through. Please help I tried everything and it was working up until my channel was migrated. What was changed?

MattiSony · 2016-03-23T09:46:31.205Z

I’m using group.tmi.twitch.tv on port 443 without any issues.

3ventic · 2016-03-23T09:51:17.292Z

You may be affected by this.

scottrc · 2016-03-23T11:58:40.182Z

will:

We’re only moving PS4 channels back, because we have a fix for them which will be deployed tomorrow (pointing irc.twitch.tv at irc.chat.twitch.tv).

Thank you for doing this

Kirawa_Akuma · 2016-03-23T17:15:19.993Z

You killed Zibrabot… I’m crying rn becasue of you! Thx! Anyway, cya! KappaPride

will · 2016-03-23T18:18:47.972Z

The last phase is now underway! I’m also updating our irc.twitch.tv DNS record to point to irc.chat.twitch.tv. You’ll see the update take effect in 15 minutes or less (900s TTL).

will · 2016-03-23T18:27:33.144Z

We’re now defaulting to AWS chat for all channels, and the DNS record for irc.twitch.tv has been updated!

Thanks for the patience and assistance, everyone! Let me know if anything comes up.

trumpi27 · 2016-03-23T18:38:09.362Z

Thanks will

Will you recommend that we remove the code that connects to the old cluster at this stage, or would you recommend that we wait a few days?

will · 2016-03-23T18:39:23.567Z

Give it a few days - we’re doing the same on our end before we shut down the old servers.

trumpi27 · 2016-03-23T18:39:58.727Z

Perfect - thank you for the quick response

IllusionaryOne · 2016-03-23T18:41:29.559Z

Now that I see duplicate messages on AWS and there were folks not receiving notifications, I think that it is safe to say that it does behave differently. There is an old saying… trust, but verify.

will · 2016-03-23T18:46:15.324Z

The message duplication was caused by a configuration issue in the testing environment. This is now resolved, and you should no longer see any double messages!

drk29 · 2016-03-23T18:49:28.185Z

Isn’t anyone really going to ask…

BUT WHY???

Why the hell do you need to create issues for bots and sub alerts out of nowhere?
How will this improve peoples experience? It’s a god damned text chat WHAT WAS WRONG WITH IT?
This message is literally “hello here are the issues we have caused”
Jesus christ what a waste of time and resources PLUS new issues for people.

moocat · 2016-03-23T18:55:09.513Z

Everything looks good on my end so far. Now running exclusively on the AWS cluster.

BarryCarlyon · 2016-03-23T19:01:33.423Z

You have no right as a third party developer to bitch and moan like this.
We were given ample warning about this, due to THIS VERY THREAD on the forums and thus had time to adapt
See 1
Twitch is concerned about it’s core product which is the delivery of Chat and Video, it is not responsible for third party tools. Where even MOST of the larger third party tools successfully whether the storm you perceive without issue.
AWS Servers for chat means that Twitch can spend less money on said servers and provide a better chat experience overall.

So stop bitching, pretty much everyone else on this thread has been constructive.

tomh117 · 2016-03-23T19:04:17.044Z

Well one of the possible benefits they listed was no more need for event servers for giant channels which alone would be really really nice for IRC users, and bots as well I guess.

And in general I would definitely call more stable/scalable/better infrastructure something that improves peoples experiences, it’s not like twitch chat didn’t have problems with that over the years (though it has gotten significantly better in the last year).

Also it’s a change that for the normal users was completely transparent except maybe some tiny hiccups, even most bot users that used the API to get the relevant room IPs shouldn’t have had any issues, it would really only affect a handful of giant bots significantly, and maybe a few streamers that had to add a .chat in a settings window somewhere.

So what I’m saying is jesus christ calm down.

tduva · 2016-03-23T19:22:41.213Z

Does it matter if we use irc.twitch.tv or irc.chat.twitch.tv now? Are both just going to stay the same or is it recommended to use one over the other?