I am doing a course in my study centre oriented to BIG DATA, I have been proposed to do a project on obtaining data in streaming form to manage it with Elastic search, store them with Logstash and display them with Kibana.
As I know this platform and I like its content I thought it would be fun to try to get streaming data from some content creator and show how many subscriptions, bits and other data he gets in his live.
Since there are streamers who can do a live stream for 12 hours or more non-stop, that data stream is continuous and I think it’s a big enough data source to do some measurement tests.
I have a short deadline and I wanted to ask you the following questions.
This is the first time I have used this type of technology and I have hardly any knowledge of it.
Could I get the data from any Twitch stream through the Twitch API to store it, filter it and display it in a graphical interface like Kibana ?
Is it possible to consume the data sets in real time with the Twitch API ?
Can I select what kind of data I want to get on a particular stream from a content creator?
Do I need some kind of access or ID of the specific streamer to get the data from his stream or can I get the data through his username or URL ?
Is the data generated by the streamers in the public domain always in accordance with Twitch’s rules and privacy?
Right now these are just some of the questions I may have about how to obtain data from a content creator during their live stream, but I’m sure I’ll have more questions or problems in the future.
This is a course project but if the idea goes ahead I could do it in a more professional way, for now I’m satisfied with passing the course.
Once again thank you for your attention.
Twitch will consider authorizing you to use the Twitch APIs to engage in the following activities only after entering into a separate agreement and running through an approval process:
Marketing - services used by marketers, promoters, or their customers for marketing campaigns, campaign analysis, or anything similar.
Research - educational and non-commercial projects that examine trends over specific time periods.
b) There isn’t enough time to get permission from interested streamers in order to collect meaninful data, especailly if you want permission from streamers that do long streams.
Then how do you expect to understand the data?
Sure collect, modify and store just like any API
Yes
Most data is scoped with the requirement to get permission from the streamer/content creator for that data.
TLDR: yes.
You need an oAuth token with the required scopes for the endpoints you wish to call/data you want to collect
No opinion offered
The biggest issue you face is getting permission from Twitch and then getting streamers permission to consume their private data.
And if you are on a short deadline as you have stated, then there likely isn’t enough time to get the relevant permission(s)
Thanks for the quick answer.
The project focused on real-time data analysis using the technologies I previously mentioned, as I had no knowledge about what guidelines I had to follow I decided to ask you.
I thought the Twitch API for this kind of activity was easier to get but to do test and simple projects.
I will then look for another dataset to analyze in my course project
Thank you for taking the time to answer me