Digital Ocean Timeouts on All API routes Kraken/Helix

[michaelwachell]

Hello all -

Experiencing connection failures/timeouts

We are experiencing an increased percentage of connection failures manifesting as TCP connection timeouts starting the afternoon of 2020-08-31 UTC. It appears to be specific to the route between SFO2 region of Digital Ocean and Twitch. We are using both Helix and v5 APIs and it appears to be affecting both. We are in contact with both Twitch and Digital Ocean support and it would help to know if anyone else is seeing similar issues whether from Digital Ocean or not. We are seeing the error r…

Exact same issue I saw (with a few others) back in the beginning of September. I went back and forth quite a bit here and with DO - I’m pretty sure this is a DO issue, but I’m curious to see if anyone else is having a solid portion of timeouts ~ 2 in 10 requests.

Both my back up server and production server started seeing timeouts exactly at 2020-11-05T14:12:27.667Z.

I’m currently running our script on a local cron and seeding our DB periodically. I’d love to hear from anyone else seeing similar issues.

[rynoranger]

Found that older post from Sept and this one… I’m also getting sporadic issues from a D.O. droplet on SFO2.

Error: connect ETIMEDOUT 151.101.190.214:443
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1141:16) {
  errno: 'ETIMEDOUT',
  code: 'ETIMEDOUT',
  syscall: 'connect',
  address: '151.101.190.214',
  port: 443,
  config: {
    url: 'https://api.twitch.tv/kraken/channels/XXXXXXXXX/videos',

Using Node + Axios 0.20.0.

[michaelwachell]

Yup, that’s the error. Thanks for posting @rynoranger. This definitely helps. If you’d open a ticket with DO as well, it might elevate our priority.

[rynoranger]

from what I can tell in past tickets is D.O. says it’s not on their end and seems Twitch isn’t really helping others figure it out.

I’m pinging that endpoint every 5 min - thought at first they were throttling the bot but the times at which it errors are very sporadic so it didn’t make sense. I’m going to hit up the twitch api discord and see if that helps.

[rynoranger]

update: got a response in discord and sent some MTR data from my droplet and hopefully that’ll help debug on twitch’s end.

Had to install these:
sudo apt install inetutils-traceroute
sudo apt install traceroute
sudo apt install mtr

Then ran this and screenshot output:
mtr 151.101.190.214 --report-cycles 1000 -r

If/when I hear anything i’ll update.

[michaelwachell]

Thanks, ryn. Did the same with DO. Hopefully we can run something down.

[michaelwachell]

[michaelwachell]

DO Stonewalling me, but the script has been running perfectly locally for the last two days - just failing on our DO prod and backup droplets

[michaelwachell]

@rynoranger - Timeouts have resolved by themselves (like last time), but I’m in the process of refactoring our core scripts to better manage them when they happen. I ended up getting zero answers. Hope you had more luck.