The partition idea could possibly open up for lots of functionality optimised for bots, without adding unnecessary processing for regular users’ chat.
Only allow PRIVMSG to channels where the bot is modded.
Only allow joining channels where the bot is modded (maybe too radical or unnecessary)
Functionality that isn’t necessarily compatible/useful with regular IRC clients, since bots with large user bases probably have maintainers who can implement it anyway.
Return useful information when reaching rate limits.
I still stand by my idea of a bot-mode where the bot is only able to talk to channels where it is modded.
The only rate limit that would be needed then is per-channel. As long as the bot only sends messages where it is allowed(modded), there shouldn’t really need to be a global rate limit, so bots don’t have to limit their user bases.
It effectively disables spam-bots in this mode.
“There’s also a per-connection server-side message queue that kicks
you out if 20 messages collect on the server, meaning that the number
of channels you can join per connection without ill effects is rather
low, especially if there’s any lag spikes”
i dont understant it. Could you explain it in other words?
“And I can count several times in the past 2 years that the site has
died because of a bug in third-party applications overloading our
servers.”
are there any public statistics for apps? Tell us which those app names
I have some input as far as joins go. https://tools.ietf.org/html/rfc1459#section-4.2.1 Using this as a base we don’t have multiple channels delimited by commas and we don’t have if invited at all.
What if there was a new scope in auth for bots in chats "ban could be useful https://github.com/justintv/Twitch-API/issues/149 ". So a bot would get bot_login instead of chat_login then when a user auths the bot they will get the chat_read.
In the end if the token has bot_login the bot can use a multiple channel join msg to the authorized channels that way we can have our own side of “important channels” and a less important list of just the users using bots via commands that don’t connect it to the api.
“nightbot and moobot etc” in this case could have their bots connections have like 80% of that connection have “important” channels fill up as fast as the server is willing to go. then they could have their current rate limited join fill that last %20 up with the regular users as they go. Crazy ideas of a man who has had little sleep. I currently have absolutely no need to join thousands of channels so this really wouldn’t do me well. White-listing seems like a good idea with the bigger bots to me.
Last thing before I go, an even crazier idea! bot_login will now produce a button in the respective bots twitch channel page. That could pop up a context menu with checkboxes over the auth scopes names and could have a button to do join and auth. This would send a CAP msg to the bots channel with “join” and the bot would join. in the browser at the same time it would auth and point back to a page redirect uri and log them into the bots webpage. Genius I know!
Has there been any changes to the connection/auth rate limiting since the outage yesterday? I’m getting “Login unsuccessful” on about 50% of my connection attempts when I’m way under the 50 in 15 threshold.
Is there any way you can keep us in the loop with status updates regarding chat when there are problems? Or, is there a more direct way to reach out when there are issues? I noticed the increase in rate limiting affected my Twitch Status site as well (making it report all servers down after the outage ended), but I had nobody to question and just assumed rate limits were silently changed as a result of the outage.
I can try to make threads during such outages here, though often during an event like yesterday we won’t be overly responsive to discussions since most of our time is spent debugging and attempting changes to resolve issues. But, I think at least a “chat is messed up. we’re doing things” followed by a “things are resolved. x, y, z changes may be affecting you” is definitely something we should do.
And just to provide some more context, we saw a 5x increase in IRC authentication / JOIN attempts, over a 2 minute period which lasted for more than an hour. We were able to drop those numbers over the 2hr downtime with some changes to our servers and clients which allowed the system to recover. We’re still trying to determine exactly what caused the increase in traffic.
I can’t really prove anything, but I would guess the extended length of outage might be caused by clients trying to reconnect (bots run a ton of connections + on-site users), while the initial spike could have been an incoming attack. If that’s the case, the earlier part of this thread where rate limits are first discussed comes to mind (since they are really the only way to prevent the total crash of the chat system). Perhaps a blanket rate limit on incoming connections and joins per server (regardless of IP) would prevent this from happening. Rate limiting is definitely an area of chat that needs work, probably sooner rather than later. If you get any progress on this front and need third party testing I’m sure myself and/or moocat would be willing to lend a hand.
Great that the rate limits are back to normal, and not permanently set to that extremely low rate limit as an “easy fix”.
Here’s hoping for a more permanent solution that gives developers breathing room but also makes sure that the stability of chat does not suffer. My numbers are showing that the explosive growth of my service definitely does not seem to be stopping any time soon.
I’m not noticing as many dropped messages, but I am noticing quite a bit of issues with chat lately. As a streamer, I wanted to know if there was anything I could do to help alleviate this.
There have been no chat issues that I have spotted in the past 24 hours. My site http://twitchstatus.com/ does a fairly good job of reporting chat status, although it can be quirky when there’s massive outages due to rate limiting.
That’s a pretty impressive site! I like being able to see the information real time and have forwarded it out to my streaming team so they can monitor as well. Thank you for sending the link.
Could I suggest you sample some low ranking guys too and see i f the same stats hold true? I constantly interact with my viewers regarding buffering (which could be their device or slow internet, not Twitch), but I also hear of people dropping chat connection all together. Not to mention my channel will go offline, but I look at OBS and I haven’t dropped frame in over 4 hours. I think a sample from the other spectrum would be nice too to see if it’s really holding or not.
I’m just a small fry at 300 follows and about 1700 views in my first 5 weeks streaming, but I just feel like Twitch is working against me sometimes.
I’ve reverted that regression on 6667 ports, creating a fix now, will push shortly. We plan to add to our http://github.com/justintv/Twitch-API docs a new section for chat, and outline our recent changes, but we want it to be in production and stable first.
Documentation for the different TWITCHCLIENT modes would definitely be handy. Every time I chat in chat my IRC client is dinging at me on the JTV mention/PM
The fix should be pushed. Unfortunately IRC clients are still affected by edge server restarts, and I had to do a couple extra because of the issue. Users watching on http://www.twitch.tv should have not noticed any of the restarts, nor dropped any messages, because we replace the IRC connection transparently.
