I’m working on live voice translation for streamers. A streamer speaks in one language, and the audio is translated in real time into another language while keeping the streamer’s own voice.
Today this works on the streamer’s side: the streamer selects our app as their microphone in OBS. What we’d like to explore is letting each viewer choose the language they hear, without the streamer needing to run a separate stream per language.
A few questions for anyone who has looked into this:
Is there any supported way to deliver more than one audio track on a live stream, with viewers picking which one they hear?
Could an Extension (for example a video overlay) play a separate audio stream in sync with the live video, or would latency and sync make that impractical?
Is there anything on the roadmap around multi-language audio, or is this something Twitch would need to support natively?
Any pointers, past threads, or people to talk to would be really appreciated.
Streamers can currently send 2 audio tracks, with 1 being the main track that all Live listeners hear, and also a separate ‘VoD Track’ which is what VoD watchers hear. This is usually used for reasons such as music that the streamer has the rights to play live, but may not have the rights to play in recorded performances (ie, VoDs). There’s no capability for users to choose which track they are listening to though, all live viewers get the live track, all VoD viewers get the VoD track.
Getting audio sync right would be incredibly tough without providing controls to the user themselves as while you may be able to get it somewhat close programmatically, even a few ms off can be noticeable so it would likely require an awful lot of effort and fine tuning for users to get right and likely not worthwhile.
Thanks Dist, this is really helpful, especially on the sync side. Good to know that before going down the Extension route.
I’ll add our use case to that UserVoice request.
One follow-up on the VOD track: if a streamer sent their original voice as the live track and a translated version as the VOD track, is there anything on Twitch’s side that would stop that from working? It seems like a way to give international viewers a translated VOD without needing new player features.
Whilst it is technically possible to for example have the live track in language X, and the vod track in language Y, I would highly discourage that as it would mean users who only speak language X would be unable to understand the VoD or any clips (as clips are based off the VoD) and can only understand live, and similarly any users who can only understand language Y would not be able to watch live or interact with chat.
What I would suggest is doing what is common in multi-language esports production and that is to stream to multiple Twitch channels. That way you can share common elements between them (such as gameplay), but give each stream not only their own language audio but it also opens up the option of different overlays and text elements on stream for each each language.
That’s a great point about clips coming from the VOD, Dist. Splitting the audience between live and VOD would hurt more than help, so I’ll set that idea aside. The multi-channel approach makes sense, and it also shows why native multi-language audio would be such a big unlock: today, reaching another language means running a whole second channel, which most smaller streamers can’t do.
