RFC 0003 - Mobile App Extensions

[splatwes]

Summary

Extensions today are usable on Desktop Web only. This proposal seeks to expand the ability for viewers to see and interact with extensions on our Mobile App.
Summary:

1. Bring extensions to a new platform by offering mobile support for extensions
2. Extensions with mobile support enabled will occupy the same area as chat in Mobile and Tablet Apps
3. Developers provide a mobile-specific front end file for their extension
4. The extension’s type on web does not impact how the extension is presented on mobile (for example a video overlay on web will show in the chat area on mobile)
5. Users can interact with one extension at a time on mobile

Motivation

While the majority of users on any given channel are on web, a large and growing number of users access Twitch via our mobile apps. Without the capability for these users to interact with extensions, broadcasters are left with the challenge of either foregoing the new interactive experiences that extensions provide or giving a confusing and lesser experience to their mobile viewers.

This represents our first of hopefully many projects on mobile. We recognize that not all use cases will be served by this initial design but our goal is to allow the majority of extension developers to start engaging mobile audiences.

Detailed design

Viewer Experience
While viewing a channel in an app, users will see an icon in the chat header to indicate that they can collapse chat.

By clicking on the arrow (iconography not final), they will be taken to a view of all extensions with a mobile view on the channel sorted by installed anchor type on web (Overlay, then Components, then Panels).

Tapping on another extension’s name will swap to that extension. Swiping left or right will also switch to the appropriate extension based on swipe direction.

Tapping on the currently selected extension’s name will bring up the extension settings menu, allowing users to share their identity or get more details about the extension.

Lastly, Twitch actions (follow and share identity) are supported and will work on mobile in much the same way as on web.

Broadcaster Setup

The broadcaster experience will be nearly identical to the experience today. The only changes are visibility changes. Extensions that support mobile will be badged specially in the Extension Manager and their details page will note that they support mobile. Specific of this badging are still TBD.

Developers, not broadcasters determine whether an extension is available on mobile. Broadcasters will not have the option to disable the mobile experience for extensions and thus impact on broadcaster setup is minimal.

Developer Setup

When configuring an extension version, a new field will be available allowing you to specify a path to your mobile viewer HTML. Presence of a valid mobile path is all that is required to enable mobile for your extension.

API and Lifecycle changes

Visibility and Extension Load
On web, whenever a user visits a page, all extensions on the page are loaded as soon as video playback begins. On mobile, this paradigm is changed. Due to constrained resources on mobile, we only load extensions as they are requested by the user. As such, until a user tries to view your extension it will not be loaded.

Your extension may be unloaded once we determine users are no longer intending to view it (typically when they have stopped looking at it for 5 minutes or longer but potentially based on other factors as well)

When a user stops viewing your extension you will stop receiving onContext updates but when the user begins viewing again you will be sent a fresh context object with the latest data.

Visibility changes can be tracked via a new function added to the JavaScript Helper:

onVisibilityChanged: function(visibilityCallback)

visibilityCallback will be fired whenever visibility changes and has two arguments

Property	Type	Description
visible	boolean	true if the extension is visible, false if not
context	object	Only given when visibility is true. The entire context object. Used to refresh your app if needed once visible

Changes to onContext
In general, we have endeavored to keep the context object unchanged but in the instances below that was not possible.

Property	Type	Description of Change
isTheatreMode	boolean	Removed for Mobile only. Theatre mode does not exist on mobile
theme	string	Added for all Platforms Indicates which theme the user is using. Valid values are light or dark

Detecting platform
For most extensions, you should be able to tell what platform you’re on simply due to the file that is loaded. If, however, you are using the same file for all platforms you can detect your platform using a new query parameter we will place on your viewer html as we load it: platform=mobile (platform=web will be added to viewer html on web for consistency)

Rollout Considerations

Adding Mobile to Existing Extensions
Developers will be able to convert all existing extensions to utilize mobile, regardless of what type of extension they are on web. The process for this will be identical to releasing a new version. No additional work needs to be done on the part of a broadcaster to upgrade to a mobile experience.

Developer Beta
We aim to give all developers access to start working on Mobile extensions approximately 3 weeks before our public launch. During this time, we hope existing extension developers will investigate and start building out mobile experiences. This will be an invaluable time for the Dev community to provide feedback on the product before we release these extensions to broadcasters.

As the developer beta begins, we will publish design and technical recommendations for building mobile extensions.

Lastly, during this period, any extension with mobile enabled will not be able to enter review but all steps prior to that will be possible.

Launch
Following the developer beta, we will launch a new version of the app with extensions enabled. Consumer adoption should be expected to be slow as is the case with all app versions.

Drawbacks

Developers will have to think responsively
Compared to panel extensions today which have static height and width, Mobile extensions will vary based on device. Rather than constraining to a specific size, we have opted to have extensions take up a percentage of screen real estate. This puts the burden on developers to present a good experience regardless of size.

Developers thinking mobile first, however, will have an easy path towards panel or component extensions on web if they develop responsively.

Alternatives

Video Overlays
After early prototyping, we determined that the experience for video overlays on mobile was not going to be the smoothest either for end users or developers. Given limited screen real estate – especially when player controls are take into account – usable screen area was quite small.

We experimented with an “interactive mode” to toggle between player and extension interactions but ultimately felt this was too much of a departure from how the app worked today.

We will likely revisit this idea in the future as users grow more accustomed to having extensions on mobile and valuable video-relative use cases continue to evolve.

[LuckyNoS7evin]

@splatwes I’d love to get in on that beta if possible, I have an overlay and panel which would be a perfect test for this. Plus I’m a “mobile first” Dev so shouldn’t be difficult (hope).

[splatwes]

Good news! We’ll be opening up the beta to all extension developers. Stay tuned for more info on dates for that.

[TheElm]

It seems optional to opt an extension into Mobile. Will it then be possible to do the opposite, and omit Web versions to create extensions exclusively for mobile users?

Before now it’s been impossible or impractical to implement Browser extensions (BTTV, FrankerZ, etc) into mobile. This seems to somewhat alter that.

A mobile mod tools extension could be created, for example, making the moderating on mobile experience better.

[splatwes]

Hey @TheElm, this has spawned an internal conversation here so I don’t have an answer just yet. Great question and I’ll post back here as soon as I have an answer for you!

[offbeatlive]

Thanks for your continued work on Extensions! I’ve enjoyed the challenge of the development process of my own Extension thus far. Here are some questions and concerns I have in response to this RFC:

1) Could you explain the UX reasoning for collapsing the chat versus any alternatives you may have explored?

@splatwes

Viewer Experience
While viewing a channel in an app, users will see an icon in the chat header to indicate that they can collapse chat.

I’m concerned for the potential need to scroll the embedded extension (provided it’s tall enough).
Vertical space can be freed up if the Stream Chat was itself positioned alongside the same menu where the Extensions are listed. Also, the input field for Chat would only appear when the Stream Chat is the selected item. Here’s a mockup of this alternative design:

This approach has advantages over the design included in your post:

• Users of Mobile Extensions have one less click/tap to navigate to an Extension.
• Extensions are discoverable by default.
• Extensions are made hierarchically equivalent to Stream Chat by being positioned in the same navigation menu (possibly leading to higher engagement as a result).
• Embedded Extensions have more vertical space to utilize.
• Fewer visual elements are needed for this design, which is already plenty dense.

2) What is the proposed design for Mobile Extensions while in Landscape?

[BarryCarlyon]

@offbeatlive

2. What is the proposed design for Mobile Extensions while in Landscape?

And is there a difference between phone and tablet dimensions

Or is tablet going to be another RFC?

[Mucx]

Thanks for the thoughtful feedback, including some wireframes to help set context of your thoughts and questions! I’ll provide some background info, and take you through our thinking in a story-format…

During our mobile redesign of last year we sought to provide clear IA/UX separation of a content creators media catalogue (i.e. Past Broadcasts, Highlights, Clips etc) and their supporting features that were previously consolidated into a single, cluttered, unfocused ‘channel page’ alongside currently playing video and more.

Through acute prioritization of user goals, we saw the creation of a dedicated playback page – aka “Theater Page” – where we surfaced contextual features or options related only to the media the user was watching in that moment. This greatly increased engagement where there were a myriad of options, buttons and navigational routes that previously proved distracting.

Extensions play into that core page goal as a related contextual experience. It was clear why and where we needed to surface this to users on mobile, but the question was ‘how’ and ‘when’ - also across mobile & tablet, portrait & landscape orientations in a way that made extension development simple.

One of the first designs my designers did was almost exactly like the wireframes above. It makes logical sense to treat chat and individual extensions as options you can navigate between while watching media. You have the benefit of seeing whats available upfront right!?

Through some simple prototypes, the team quickly realized that having chat, and extensions on the same plain “just…ehhh, didn’t feel right”. Digging into this, and questioning (internal/external users) we started realizing that the current batch of extensions where fantastic material to glance at, to have on-hand when needed, but the core focus of their time was still watching video and engaging in chat.

I dubbed this “the 2nd monitor paradigm”. Putting chat at the same level as extensions, such as in a tab construct, caused a lot of navigational friction when users wanted to multi-task between, to “glance”, or quickly interact with an extension before settling back into video + chat. (i.e. I want to see the extension in tab 6, and then go back to chat - how can I do this quickly?)

That is where the collapsing ‘chat toolbar’ flow came to be. It allowed for easier multi-tasking. You can bounce quickly back-n-forth between chat and an extension. In fact, you can ‘park’ an extension (it remembers the tab you had focused) making this multi-tasking even speedier between the two.

Now, this is a big bet on user behavior, but we have modeled it on user journeys/flows we have witnessed and the types of extensions being developed (you’ve all done some cool work!). As you guys test the boundaries of extension interaction, different extension types, and as we learn more from the user base through their usage (quantitative and qualitative) we will of course adapt the UI and how users watch media and interact around it.

[Mucx]

Also, here are the phone landscape and tablet landscape designs below. We have sought to make sure extensions developed for mobile “just work” or you can design and build once and feel confident in them working across devices and their orientations with minimal fuss (so long as they are responsive & don’t use fixed widths)…

As you can see, for landscape phone, when invoked the extension area expands slightly to accommodate the extension and tab construct. (Static images don’t give you a sense of this nice fluid animation at play here).

On tablet, it works similarly. (Keeping some chat visible is still TBD)

One of the goals for not only extensions developers but users is having a familiar UI across devices. Learn how to use and develop once!

[Diogee]

@TheElm

When we created extensions, our goal was (and is!) to enable new types of shared interactive experiences between broadcasters and their communities. We’re bringing extensions to mobile as a step to unify the customer experience across all platforms, so regardless of where you watch, you can participate in extension content.

You bring up a great point about an extension being able to add moderation tools to the mobile experience. It appears, as you describe it, you already have a web version of that extension active (BTTV or FFZ), it’s just not a Twitch Extension! I believe the way we’ll solve for your concern long term is to create more flexible functionality on web such that an improved moderation tool Twitch Extension makes just as much sense on web as it does on mobile and you have access to those tools regardless of platform in a fully supported way.

For now, we won’t be adding the capability for mobile-only extensions.

Thanks for the feedback!

[offbeatlive]

Likewise, thanks for the detailed response! This is pretty much exactly what I’d hoped to hear back. Getting a glimpse of the process behind making Extensions gives me much more confidence in choosing to develop one.

I will say, my original concern still stands. It seems to me that the UX route you and your team have decided on is just as potentially effective as the alternative. Like you say, it’s a bet on user behavior, the reality of which is unknown until Extensions are available on mobile. Still, I respect the process taken to make that decision. My lingering apprehension comes from a couple doubts:

1) How many tabs are reasonably expected to be the norm?

You used the example:

I want to see the extension in tab 6, and then go back to chat - how can I do this quickly?

Is it likely that 5 mobile extensions would be norm, or even more than say, 3 extensions? The speed gains of the presented design are only present if, in the alternate design, the tab being switched to is only accessible by scrolling the menu. If the target tab is adjacent to the Stream Chat tab, switching between them is just as convenient as collapsing the chat would be. The key difference between the two approaches, then, would be what I consider the main advantages of the alternative design: preserving vertical space and reducing visual density.

2) The open-ended possibilities of Extension design

A partial basis for choosing the presented design is that:

…the current batch of extensions where fantastic material to glance at, to have on-hand when needed

Other current and potential future Extensions aren’t necessarily characterizable as reference material, so designing for that type of interaction might cuff other Extensions. In the case of an Extension which accepts keyboard input, I’d argue that it’s best to preserve as much vertical space as possible. Mobile viewports are cramped with on-screen keyboards as it is. Consider also an Extension which does supplant chat, yet still keeps the user engaged with the video content. Wouldn’t a fixed navigation element to return to chat become a distraction?

This is speculative, of course, yet still very possible within the existing constraints. Due to how open-ended the possibilites for Extensions are, my only bid here is to make a case for the following: give Extensions as much space as possible to allow them to fully engage the user and succeed (or fail) upon their own merit. I think the alternative design satisifies those conditions more than the presented design.

---

All this being said, it’s clear to me that building the Extensions platform is no trivial task. As far as I can tell, configuring a successful platform for Extensions is like trying to solve a multivariate equation, and in this case, none of the variables have settled into even a remotely stable value. I would like Extensions to succeed, so I’m offering my criticism here to increase the chances of that happening.

Tons of gratitude and appreciation are due to you and your team for creating this opportunity in the first place, so thanks!

[BoldBigflank]

Ok, so you can switch between chat and the extensions quickly. Could there be a way for an extension to call attention to itself? Either a notification icon or even collapsing chat to show itself. Imagine a poll extension that is only needed when a poll is started, and needs input quickly, but is not worth glancing at until a specific time.

[splatwes]

We’re toying around with something just like that but it’s not going to make it into v1. Totally agree this is valuable and a polling extension is a perfect example.

[Seravy]

I am keen for this change.

To be honest the thing that excites me the most is the theme property on the onContext as that’s been a real sticking point!

As for the mobile changes, they all look good. To date i have had to run a parallel web site so people who use mobile and desktop twitch app can still participate.

Cheers

[BoldBigflank]

I see that the extension settings page has the “Mobile” checkbox and file input. Is there more info on the beta test of this? How would I test the mobile html on an app?

[Diogee]

Hey @BoldBigflank,

We’re in the process of preparing the dev site for mobile extensions availability. Keep your eyes peeled here and on our Twitter for additional information on when mobile extensions are available for initial development Soon ™!

[bribrobot]

Exciting, think you’ve found a really nice design solution for mobile!

It’s maybe already planned, but wanted to call out that (1) a set of browsers to make sure to test on (Chromium-like? Safari Mobile? Android browser?) & (2) dimension ranges to support would be super helpful to have in the docs for development purposes!