STORIES
This blog is written by Sjim Wagemakers, Tech Lead smart TV Development at Triple.
In the world of software and app development, challenges are often considered obstacles. However, I feel that these challenges should be seen as opportunities for growth and innovation, really. As often, these very challenges actually keep things interesting, even if you have been doing this work for years. At Triple, we develop apps for many different cool clients like NLZIET, NOS, KIJK, NewsON, VTMGO and STREAMZ. For these clients, we work with several different platforms. While we mostly focus on WebOS (used on LG) and Tizen (used on Samsung), we also support AndroidTV, Amazon FireTV, Hisense, VEWD, and even Roku!
You might think that smart TV development is a niche profession, working in some obscure programming language. But in real life, we are part of the cool kids gang. We get to use modern HTML, CSS and Javascript most of the time (sneak peek, more on that later!).
So actually, developing smart TV apps is awesome. In this article, we will talk about some of the challenges that add some extra spice to our work.
Challenge 1: Performance
Fun fact: contrary to their name, smart TVs are often actually not that smart. Most consumers are not that eager to spend large amounts of money on their TV, so manufacturers tend to cut costs on hardware for popular models. After all, the most important feature of televisions is to play video and to do that, it only needs basic hardware. Developing apps for smart TVs is therefore not always easy, but we know exactly how to deal with this challenge.
Top-down approach
At Triple, we work with a top-down approach, meaning we create apps based on the very best hardware specs with ultimate capabilities. If a device range does not support the technique we have chosen for a feature, we make adjustments for that specific range. Through this approach, we can leverage the full potential of the available hardware for newer devices, which leads to a smoother and more immersive user experience. It does implicate that we need to know where the limitations and constraints for lower-end devices are, so we constantly keep an eye out for where this might impact the user and update our applications accordingly.
Svelte
In order to get the best performance out of our smart TV applications, we wanted to choose a JavaScript framework that is truly focused on performance, which is why we chose Svelte as our default JavaScript framework. It is lightweight and shows great performance, even on lower-end devices. Svelte is often used on devices with limited hardware, such as point-of-sale terminals. As it turns out, for this reason, Svelte is also a perfect fit for use on smart TVs.
Where most JavaScript frameworks are shipped with the bundle, Svelte does some magic in the compile step and outputs highly optimized and efficient code. Code compiles (ahead of time) to stand-alone framework-less, pure JavaScript that manipulates the DOM directly, so no JavaScript framework needs to be included in the bundle. This results in smaller bundle sizes and improved runtime performance.
Besides great performance for users, Svelte also offers a better (if not the best) developer experience 😀. From the ‘State of JS 2022’, a survey filled in by thousands of Frontend Developers, it became clear that Svelte usage is steadily growing and has second-to-best retention. An increasing number of developers are working with Svelte and when they do, they have the intention to keep using it. Needless to say, we share their enthusiasm.
Working with web technologies has an additional advantage: we are able to work locally in the browser. Of course, we can only do this up until a certain stage of development, as to ensure everything works perfectly, it has to be tested and checked thoroughly on the TVs as well.
Challenge 2: Input
As handy as they may be in usage, remote controls restrict the complexity of user interaction in smart TV apps. While a mouse and a keyboard have numerous options for advanced user interaction, a remote control only has a limited number of buttons, that are often allocated to specific functions like navigating menus or basic actions. This makes additional commands like performing precise movements or giving text input simply harder to issue. As a result, when developing smart TV apps, we often need to re-think things that are non-issues on websites, like navigation.
Spatial navigation
Spatial navigation refers to moving between focusable elements in a user interface. It simplifies the process of exploring and selecting options on the screen, making it more user-friendly for everyone, regardless of their technical knowledge. If this technique performs well, it should feel quite simple to users. But ‘under the hood’, it is all but simple. Spatial navigation is a very complex technique with a lot of edge cases.
Let’s see how this works. Imagine we focus on one element, let’s say a card in a row with videos, and the user presses the ‘UP’ key. The library will check the DOM to identify any elements positioned directly above the current item, it will try to determine which one is closest and focus on that. But what if there is no item directly above it? In that case, the block positioned above it can be regarded as one whole element, we can determine what item in that block is closest to our current item and focus on that specific part.
So, the navigation works quite differently than it would with a keyboard and mouse. During development, this means that we need to think differently as well. Some LG devices use a remote control that has a Gyroscope, called the “Magic remote”. This enables users to move a cursor on the screen (like a regular mouse on your computer), by moving their hand and clicking on things. When using this feature, elements on the screen also need a ‘hover’ state, an indication of where the mouse is currently at. There are also smart TVs on the market that offer touch support. All of this means that we must support spatial navigation as well as clicks, scroll and touch. To offer smooth and ‘as painless as possible’ navigation, we have built our own Spatial Navigation library. Because in a niche field like smart TV apps, we just need to think outside the box.
Challenge 3: Devices
When it comes to smart TV, there are many different devices on the market. And they all work differently. For this reason, we have quite a few of them in our test lab at Triple, including LG, Samsung, Hisense, Philips, Roku, Fire TV and many more. This causes some practical issues as well; because remote controls are usually universal to all models of their brand, you can easily end up accidentally controlling multiple TVs at once. This is why in our office, we separate different TVs of the same brand as much as we can. We have also implemented some creative, non-tech solutions like putting a toilet paper roll over the remote, to be able to properly focus the infrared signal to the correct TV. Slightly unorthodox, but highly efficient 😊. Some devices are controlled via an app, so fortunately we don’t need a roll for every remote control.
Balancing between features, limitations and opportunities
During development, we need access to all these different TV models because of the many differences. Most of them use an older browser version and support for CSS and JS heavily varies. For instance: when we work on a ‘simple’ card component, everything can look fine and dandy in the web browser and on most TVs. But there might be one specific older model TV that has a slightly different positioning or scaling for this element. The only way we would know is by testing features on every supported TV. Although this is just an example, cases like this are actually quite common.
From experience, a developer generally knows which techniques might be risky to use, and which techniques are widely supported. Our top-down approach means we always keep an eye out for features that are supported by newer devices, so we can offer the best and most up-to-date experiences on those devices. Of course, in practice, we end up elaborately testing on the worst and oldest devices, as they will have the most limitations.
Available tools
As smart TV developers, we do not always have the same tools available as web developers, so we need to be resourceful. For example: when developing websites, there is an awesome tool, caniuse.com, which shows you all the differences between browser versions. Sadly, this type of reference does not exist for smart TVs, so we have created a similar overview ourselves. Obviously, it is not as extensive as caniuse.com, but it is very helpful, speeds up development and reduces bugs in the apps we build.
Luckily, developer tools are available through all devices in one way or another. They are generally offered in an old version of Chromium, meaning we can’t use the latest features of modern browsers. We do have the ability to remotely debug, inspect elements, debug JavaScript code, analyze network activity, profile performance, and track errors through the console; similar to the process of regular website debugging. Having all these tools available is key to a debugging workflow and accelerates issue resolution. So even though we have to use workarounds sometimes, we make sure the end result is always optimal.
Challenge 4: Working remotely
During the Covid pandemic lockdowns, we were forced to work from home. As smart TV developers, this was challenging, to say the least. None of our homes have space to fit a dozen TVs comfortably, but we do need these devices almost every day. To address this issue, we came up with a creative solution (even more creative than the earlier mentioned toilet rolls on our remotes😊). We installed two cameras in our test lab that can be controlled remotely and can rotate 360 degrees. This allows us to control the cameras from home and point them at any TV we need.
Of course, controlling the TV is a second challenge. For this, we have installed a system with an infrared sensor below each tv. This sensor can be controlled from the internet via a web app or a mobile app, giving us remote control over the TV.
Although this system is not perfect, it has allowed us to continue our work and stay productive. We hope to continue refining and improving our remote work setup as we maintain a flex-working policy going forward, and our developers can work from home a few days a week.
Challenge 5: Streaming video
The video player is definitely one of the most important elements in our apps, the user experience and all of our challenges. After all, video is what apps like Netflix (or NLZIET or NOS) are all about.
If you look at the HTML video element, playing a video seems simple. Just throw in some source video files, maybe even multiple files for full support and that’s that. Right? You would think so. Unfortunately, there are some pitfalls. Generally, the whole file needs to be downloaded to play. While this works fine for smaller files, it does not for 2-hour videos. And this is where streaming video comes in.
Streaming solutions for optimal experience
To improve the experience, a good solution is ‘progressive streaming’, where the metadata is located at the very start of the video file. This means the video can start playing while it is still downloading the rest. But with progressive streaming, we still work with a single video file and format for all devices. For a big screen, the quality might be too low, while it is too high for a very small screen. If you would want to skip parts of the video, you will have to download the whole file up until the point you want to skip to. If you have a low-quality internet connection and the video data is not loaded fast enough, the video will have to pause and wait for more data (‘buffering’). And we don’t know about you, but waiting is not our favorite pastime.
With video streaming, we split a big video into multiple smaller pieces called ‘segments’, and they are served one by one. This way we can only load the parts of the video that we actually need. So, if we move to a point halfway through the video, we only need to download the segments near that point and continue from there. We use a playlist to determine what segment goes where, so the player knows what to download and play.
Adaptive Bitrate Streaming (ABR)
Adaptive Bitrate Streaming (ABR) is a sophisticated technology that dynamically adjusts the quality of video content in real time, ensuring smooth playback regardless of network conditions.
🤔 What is a bitrate?
Bitrate is the amount of data processed per unit of time. i.e. Bits per Second. Generally, a higher bitrate means better quality, but also a bigger file size. In adaptive bitrate streaming we also have multiple files (segments) in a stream, but now also at different qualities (or bitrates) for different purposes (like screens/devices). And the best part: a player can adapt the selected quality to the available internet bandwidth, and seamlessly switch between them, automatically or by the user’s choice.
Let's illustrate with an example: imagine watching a video on your mobile phone in the back of a moving car. Your connection heavily depends on your location. When you start watching the video, the connection might be very good and you can enjoy the highest-quality video files. But when the car enters a tunnel, the internet connection might drop. And here comes the trick: when the video player is unable to download the video frames fast enough to offer a smooth experience, it will automatically switch to a lower quality. The user might notice the drop in quality, but this experience is way better than pausing the video to buffer.
Video streaming is a complex process, due to its time-based nature and the many multiple processes that occur simultaneously. We need the video player to deliver the content seamlessly for an optimal experience. Therefore, video streaming technology is a crucial element for us, as we need to make sure that users have a consistent and uninterrupted viewing experience.
The pros and cons of HTML video tags
Before the HTML video element was introduced in 2007, we used programs like QuickTime or Shockwave to display video on websites. They felt alien to the webpage, simply because they were. Using 3rd party plugins is like punching a hole in the webpage and putting a video player behind it. As these were third-party plugins, the players themselves were a black box, and we did not have control over what would happen. When Opera proposed the HTML video tag in 2007, fortunately, everything changed.
The HTML video tag is the opposite of a black box. You can even right-click to download the video file (which could potentially be a problem when it comes to the copyrights to that video.)
Digital Rights Management
And this brings us to Digital Rights Management (DRM), which makes things even more complicated…
From the videotape era, you might remember the messages at the beginning of the videotape asking you ‘nicely’ not to copy it. With DVDs, creators stopped asking nicely and encrypted the video content. DVD players contained the key to decrypt the content. It might not have been the most secure system, but it worked at the time. Nowadays, things have gotten a lot more intricate.
Different services for different devices
In today's digital landscape, the protection of intellectual property and copyrighted content is of utmost importance. Digital Rights Management serves as a crucial tool in ensuring that digital content remains secure and protected from unauthorized access or distribution. DRM contains piracy prevention, access control and usage restrictions, all through encryption. However, encryption alone is not sufficient. If the decryption key is transferred openly, and readable to the client, it would offer very limited security. This is why there are services that obscure the key: “Security by obscurity”.
When we talk about DRM in streaming, we are usually referring to three different services:
Widevine - By Google
Fairplay - By Apple
Playready - By Microsoft
As different devices offer support for different services, using only one service generally does not give you sufficient coverage. If you want an app to be available on all devices, you might need all services.
How does a service like this work?
The video player receives video content from CDN, which could be encrypted.
The playback application sends a request to the DRM license server. This part is handled by a content decryption module, which is a piece of software embedded in the browser or device. This means that we, as developers, have no control over it.
The license proxy server acts as an intermediary between the playback application and the license server. It helps manage license requests, adds an additional layer of security, and provides scalability benefits.
The license server verifies the request and generates a license containing decryption keys and usage rights specific to the user and their device.
The license is sent back to the playback application on the user's device and can include a set of instructions like expiration dates, what device it can be played on, or if it can be downloaded.
The player uses the received license to decrypt the encrypted content, making it playable.
Once decrypted, the content can be securely played back on the user's device, ensuring that it is protected against piracy and unauthorized copying.
Of course, this is just a general idea of the process. While we know enough to work with it from our (the player) side, we don’t need to know every last detailed step involved. And that is exactly the idea behind DRM services: a black box, security by obscurity.
Conclusion
In conclusion, smart TV app development presents a unique set of challenges that developers must overcome to deliver a seamless and engaging user experience. Overcoming these challenges not only requires a combination of technical expertise, thorough planning, and effective collaboration, but also resourcefulness and creativity. We need to think outside the box. And this is precisely what makes developing smart TV apps so interesting.
Ultimately, successful smart TV app development relies on understanding the unique challenges of the platform and leveraging innovative solutions to deliver high-quality apps that provide users with an immersive and satisfying entertainment experience. At Triple, this is what we aim for. And preferably, a rewarding and enjoyable experience for developers as well 😊.
Explore the Triple Universe
Learn more about Triple, our culture or check out our vacancies.
