Architecture of a video on-demand service that uses Azure Media Services

Previously I’ve explained that Azure Media Services provides everything you’ll need to build and operate video on-demand services to multiple devices and platforms, including all the tools and services you’ll need to handled media processing, delivery, and consumption.

In this blog post I’ll examine the architecture of a video on-demand service created with Media Services for the Building an On-Demand Video Service with Microsoft Azure Media Services project. As the guide focuses on the Windows Store app developed for the project, I’ll focus on the Windows Phone app.

Understanding the video on-demand service architecture

Building and hosting a video on-demand service can be a major undertaking and can require a significant investment in hardware, management, and other infrastructure resources. Connectivity and security are also major concerns because users require timely and responsive access, and at the same time the system must maintain the integrity of the data and the privacy of users’ information. To support a potentially large number of concurrent users against an ever-expanding video collection, we chose to implement the video on-demand service by using Azure Media Services. Media Services allows you to build scalable, cost effective, end-to-end media distribution solutions that can upload, encode, package, and deliver media to a variety of devices and platforms. In addition, the Azure environment provides the necessary scalability, reliability, security, and performance necessary for supporting a large number of concurrent, distributed users.

The following diagram shows a high-level overview of the video on-demand service.

Client apps communicate with the video on-demand service through a REST web interface. This interface allows the apps to retrieve, upload, and publish videos. When a video is uploaded for processing by Media Services it is stored in Azure Storage, with the video details being stored in a Content Management System (CMS). It’s then encoded to a set of adaptive bitrate MP4s, which can be converted by dynamic packaging to smooth streaming, HLS, or MPEG-DASH, on demand.

When a video is consumed by apps, its URL is retrieved from the CMS and returned to the app. The app then requests the URL content from the Media Services Origin Service, which processes the outbound stream from storage to client app.

There are three main components to the video on-demand service:

• The client apps, which allow users to browse videos and control the playback of videos. In addition the apps allow users to capture and upload new videos for encoding by Media Services, which can then be consumed from the CMS once encoding has completed. This series of blog posts will focus purely on the Windows Phone app.
• The business logic, implemented as a web service. This web service exposes the data and operations that it supports through a REST interface. Separating the business logic in this way decouples it from the client apps, minimizing the impact that any changes to the implementation of the apps will have on this business logic.
• Data storage, provided by an Azure SQL database, and by Azure Storage. The CMS, which stores details of videos and encoding jobs, is implemented as a Azure SQL database. However, uploaded videos and encoded videos output by Media Services are stored in Azure Storage.

Understanding the Windows Phone app architecture

We used a modified version of Prism for the Windows Runtime to build the Windows Phone app. Prism includes components that provide support for MVVM and the core services required in Windows Phone apps.

The following diagram shows the architecture of the Windows Phone app and the web service in more detail. Grey items are provided by the modified version of Prism for the Windows Runtime, with blue items being created by the development team.

The advantage of this architecture is that it helps to produce flexible, maintainable, and testable code, by separating the concerns of presentation, presentation logic, and entities through support for MVVM.

The Windows Phone app uses the Unity dependency injection container to manage the instantiation of the view model and service classes in the app. Dependency injection enables decoupling of concrete types from the code that depends on these types. The UnityContainer object holds a list of registrations and mappings between interfaces and abstract types and the concrete types that implement or extend these types. The App class instantiates the UnityContainer object and is the only class that holds a reference to this object. Types are then registered in the InitializeContainer method in the App class. For more information about using Unity, see Unity Container.

Summary

This blog post has introduced the architecture of the video on-demand service, and the Windows Phone app that consumes the video on-demand service through a REST interface. Media Services is used to encode and package video into the required formats for consumption across a variety of platforms and devices. For more information see Building an On-Demand Video Service with Microsoft Azure Media Services.

In my next blog post I’ll discuss the development of the CMS which stores details of videos and encoding jobs.

NDC magazine article on Prism for the Windows Runtime

NDC Magazine is a developer magazine written by developers, and aims to cover the latest trends within software development and technology. It has a circulation of 12,000 magazines that are distributed to developers and decision makers in the software industry. As well as a print edition, it also has an online presence at www.ndcmagazine.com.

The latest edition of NDC magazine is now available, to coincide with NDC Oslo 2014 in the first week of June, and features an article about using Prism for the Windows Runtime to accelerate the development of Windows Store apps, written by myself.

You can download the PDF edition of the magazine by clicking on the image below.

Azure Media Services Overview

Previously I’ve mentioned that I’ve worked on the recently released Building an On-Demand Video Service with Microsoft Azure Media Services guide, created by patterns & practices. In this blog post (the first in a series about Azure Media Services, which will focus on the Windows Phone client we created) I’ll provide an overview of the functionality and capabilities provided by Azure Media Services.

Traditionally, building the workflow for the creation, management, and distribution of media is problematic. It involves having to integrate multiple technologies and providers, some of which may be incompatible. In addition, it can require a huge investment in infrastructure, which may not always be fully utilized. These issues can result in a non-standardized workflow that is not easily scaled, and that requires coordination at different stages of the workflow.

Azure Media Services allow you to build scalable, cost effective, end-to-end media distribution solutions that can upload, encode, package, and stream media to Windows, iOS, Android, Adobe Flash, and other devices and platforms. The advantages of Media Services over the traditional approach to building a media workflow include:

• An API that allows you to easily create, manage, and maintain custom media workflows.
• A standardised workflow that improves coordination and productivity when there are multiple participants involved in the creation and management of content.
• Automatic scalability by using global data centers to transcode and deliver media assets, without having to plan for capacity spikes or worry about idle data centers.
• Cost effectiveness by encoding media once, and then using dynamic packaging to deliver it in multiple formats.

The diagram below shows a high-level overview of the standard workflow used when processing media with Media Services.

The steps involved in the workflow are as follows:

1. Media is uploaded to Media Services and stored in Azure Blob Storage.
2. Uploaded media is encoded using the Azure Media Encoder, with the encoded media being stored in Azure Storage.
3. Encoded media is packaged by the Azure Media Packager, with the result being stored in Azure Storage.
4. Client applications playback the media located at a URL, with the Origin Service processing the outbound stream from storage to client application.

I’ll now briefly discuss each item in turn.

Uploading media to Azure Media Services

You must upload your content into Azure Media Services in order to be able to encode, manage, and consume it. Media Services uses Azure Storage to store your media for processing and viewing. Your content can be programmatically uploaded using the Media Services REST API or one of the available client SDKs. These APIs allow you to upload one file at a time or perform bulk upload operations. Media Services also allows you to perform secure uploading and storage of your content. Storage encryption will encrypt your content locally prior to uploading it to Azure Storage where it will be stored in an encrypted form.

Supported file types

Various video, audio, and image file types can be uploaded to a Media Services account, with there being no restriction on the types or formats of files that you can upload using the Media Services SDK. However, the Azure Management portal restricts uploads to the formats that are supported by the Azure Media Encoder. These import formats include MPEG-1, MPEG-2, MPEG-4, and Windows Media Video encoded video, MP3, WAVE, and Windows Media Audio encoded audio, and BMP, JPEG, and PNG encoded images. The Azure Media Encoder can export data as Windows Media Video, Windows Media Audio, MP4, and Smooth Streaming File Format.

Processing media with Azure Media Services

In order to process media with Media Services you must obtain a Media Processor instance, prior to encoding, packaging, and protecting media files.

Obtaining a media process instance

Media Services provides a number of media processors that enable video to be processed. Media processors handle a specific processing task, such as encoding, format conversion, encrypting, or decrypting media content. Encoding video is the most common Media Services processing task, and it is performed by the Azure Media Encoder.

Encoding media

Encoding is the process of taking a video and turning it into a format that can be consumed by users. Users could be using a variety of devices to watch your videos, including desktop computers, smart phones, tablets, Xbox consoles, set-top boxes, or Internet-connected TVs. These devices all have features that affect the required encoding. For instance, smart phones have small screens and little storage, while desktop computers have larger screens and larger storage. In addition, smart phones potentially have a more limited bandwidth than desktop computers. Therefore, when you choose how to encode a video you must bear in mind the variety of devices that users will consume the video on.

The Media Encoder is configured using encoder preset strings, with each preset specifying a group of settings required for the encoder. Encoder presets are divided into two groups – general presets and device specific presets. Videos encoded with general presets can be used by any device that supports the required file formats. Videos encoded with device specific presets are designed to be used by a specific device, such as a smart phone.

Packaging media

Once a video has been encoded it is placed in an output asset, which can then be placed into a variety of file containers. This process is referred to as packaging. For example, you could convert an MP4 file into smooth streaming content by using the Azure Media Packager to place the encoded content into a different file container.

Media Services allow the user to decide if they will package video upfront with a media processor, known as static packaging, or package video on demand, known as dynamic packaging.

Protecting media

To protect your media when it is published, Media Services supports PlayReady sample-based Common Encryption and AES 128-bit CBC Envelope Encryption. For more information about content protection see Protecting Assets with Microsoft PlayReady.

Delivering media from Azure Media Services

Media Services provides different mechanisms for delivering media assets that have been uploaded to Media Services. There are typically four approaches that users can use to access videos:

1. Offline viewing involves a user downloading media onto their device.
2. Progressive downloading allows a user who is connected to the internet to start viewing a media item before it has been downloaded.
3. Streaming requires an internet connection but only downloads a small amount of the media item at once and discards it once it has been viewed.
4. Adaptive bitrate streaming allows client applications to determine network conditions and adapt the data rate of the media content to the available network bandwidth. Media Service supports Smooth Streaming, HTTP Live Streaming (HLS), and MPEG DASH.

Accessing content in Media Services requires a locator, which provides an entry point to access media files. An access policy is used to define the permissions and duration that a client has access to a media asset. There are two types of locators:

1. Shared access signature locators grant access rights to the underlying blob container of a media asset in Azure Storage.
2. On-demand origin locators grant access to streaming content through the Origin Service, which pulls the content from Azure Storage and delivers it to the client.

Consuming media from Azure Media Services

Media Services provides support for creating media player applications that run on different devices and platforms including PCs, Macintosh, Windows Phone, iOS devices, and Android devices. Microsoft also provides many different SDKs and player frameworks that allow you to create applications that consume streaming media from Media Services. For more information see Developing Azure Media Services Client Applications.

Summary

Media Services provides everything you'll need to build and operate video-on-demand services to multiple devices and platforms, including all the tools and services you'll need to handle media processing, delivery, and consumption. In addition, Media Services will help your platform scale by using the global footprint of Azure datacenters, without having to plan for capacity spikes or worry about idle datacenters. Together, this helps to reduce the costs that are associated with integrating multiple products and providers when building a media solution. For more information see Building an On-Demand Video Service with Microsoft Azure Media Services.

In my next blog post I’ll examine the application architecture of a video on-demand service created with Media Services.

Data virtualisation using the ISupportIncrementalLoading interface

Sometimes the data set an app works with is so large that it shouldn’t be stored in memory. For example, if your photo collection contains tens of thousands of photos it is not sensible to load all the photo thumbnails for display when the page loads. This is because it will require a substantial amount of memory to store the data that’s being displayed. Generally, high memory use degrades the experience for all apps on the system. In addition, the chances of a suspended app being terminated rises with the amount of memory being used by the active app. This is also true for the chances of the app being terminated when it’s inactive. In such circumstances you should implement a data virtualization strategy, where you load an initial portion of the data set into memory, and then load further data incrementally on-demand.

Incremental data virtualization is one such strategy and sequentially retrieves data. For example, a GridView that uses incremental data virtualization and contains 10,000 items could choose to retrieve only the first 20. Then the next 20 items can be retrieved as the user pans through the grid. Each time more items are retrieved the scroll bar for the grid becomes smaller. To implement this type of data virtualization you must use a collection that implements the ISupportIncrementalLoading interface. This interface defines one method and a property:

• HasMoreItems is used to determine if the collection has more items available to be retrieved.
• LoadMoreItemsAsync is invoked every time the control that binds to the collection needs to retrieve more items for display.

By implementing this interface, if data is being displayed in a GridView, when the scroll thumb reaches the edge of the screen, after having checked the HasMoreItems property, the LoadMoreItemsAsync method is invoked to retrieve more data for display.

Implementation

In order to implement incremental loading we must have a class that inherits from the ObservableCollection<T> class, and which implements the ISupportIncrementalLoading interface. The IncrementalLoadingPhotoCollection does this.

public class IncrementalLoadingPhotoCollection : ObservableCollection<IPhoto>, ISupportIncrementalLoading

{

    public bool HasMoreItems { get; private set; }

    public IncrementalLoadingPhotoCollection(IRepository repository)

    {

        ...

        HasMoreItems = true;

    }

    public IAsyncOperation<LoadMoreItemsResult> LoadMoreItemsAsync(uint count)

    {

        return InnerLoadMoreItemsAsync(count).AsAyncOperation();

    }

}

The IncrementalLoadingPhotoCollection constructor sets the HasMoreItems property to true, indicating that there is data which can be incrementally loaded. The LoadMoreItemsAsync method will be automatically invoked to load data for display on the MainPage. It does this by invoking the InnerLoadMoreItemsAsync method.

private async Task<LoadMoreItemsResult> InnerLoadMoreItemsAsync(uint expectedCount)

{

    var actualCount = 0;

    ObservableCollection<IPhoto> photos;

    try

    {

        photos = await _repository.GetPhotosAsync(_photoStartIndex);

    }

    catch (Exception)

    {

        HasMoreItems = false;

        throw;

    }

    if (photos != null && photos.Any())

    {

        foreach (var photo in photos)

        {

            Add(photo);

        }

        actualCount += photos.Count;

        _photoStartIndex += (uint)actualCount;

    }

    else

    {

        HasMoreItems = false;

    }

    return new LoadMoreItemsResult

    {

        Count = (uint)actualCount

    };

}

This method creates a new ObservableCollection of IPhoto and calls the GetPhotosAsync method of the FileSystemRepository class to retrieve the first 20 photo thumbnails. Provided that data has been retrieved, it’s then added to the IncrementalLoadingPhotoCollection for binding to by the MainPage view. When there’s no more data to retrieve an exception occurs and the HasMoreItems property is then set to false, indicating that there’s no more data to retrieve. A LoadMoreItemsResult structure is returned, which wraps the asynchronous results of the LoadMoreItemsAsync call. This structure contains a Count field, which indicates the number of items that were loaded.

The GetPhotosAsync method in the FileSystemRepository class is responsible for retrieving up to 20 thumbnails on each invocation.

public async Task<ObservableCollection<IPhoto>> GetPhotosAsync(uint startIndex)

{

    ...

    var files = await fif.GetFilesAsync(startIndex, _defaultPageSize);

    ...

}

The first time GetPhotosAsync is called it retrieves the thumbnails for the first 20 photos on the file system (index 0 to 19) by calling one of the overloads of the GetFilesAsync method. The _defaultPageSize field is set to 20, indicating that a maximum of 20 FileInformation objects will be returned. Therefore, the next time the GetPhotosAsync method is called it will retrieve the next 20 photo thumbnails (index 20 to 39), and so on. The starting index for each call to GetPhotosAsync is stored in the _photoStartIndex field in the IncrementalLoadingPhotoCollection class.

The GridView on the MainPage binds to the Photos collection in the MainPageViewModel class. The Photos collection is of type IncrementalLoadingPhotoCollection.

public IncrementalLoadingPhotoCollection Photos

{

    get { return _photos; }

    private set { this.SetProperty(ref _photos, value); }

}

When the MainPage is navigated to, the OnNavigatedTo method in the MainPageViewModel class will be executed. This simply initializes the Photos property to a new collection of type IncrementalLoadingPhotoCollection.

public override void OnNavigatedTo(object navigationParameter, NavigationMode navigationMode, Dictionary<string,object> viewModelState)

{

    base.OnNavigatedTo(navigationParameter, navigationMode, viewModelState);

    Photos = new IncrementalLoadingPhotoCollection(_repository);

}

The overall effect is that when the MainPage is loaded a subset of your photo thumbnails are retrieved from the file system for display in a GridView. When you you pan through the GridView more photo thumbnails are incrementally retrieved from the file system on-demand for display. This avoids having to store all the photo thumbnails in memory, thus decreasing the memory footprint of the app and increasing the experience for all apps on the system.

Summary

This blog post has extended the sample app by implementing data virtualization using the incremental loading strategy. This was achieved by creating the IncrementalLoadingPhotoCollection class, which implements the ISupportIncrementalLoading interface. This class uses the FileSystemRepository class to retrieve 20 photo thumbnails at a time, and incrementally retrieves further thumbnails on-demand as the user pans through the GridView that displays the thumbnails. This avoids having to store all the photo thumbnails in memory, thus decreasing the memory footprint of the app and increasing the experience for all apps on the system.

The sample code can be downloaded here.

Building an On-Demand Video Service using Microsoft Azure Media Services

The patterns & practices group at Microsoft have just released a guide and reference implementation for a new project called Building an On-Demand Video Service with Microsoft Azure Media Services. As a team member of the project, and someone with an interest in the unique challenges of building video-on-demand services, I thought I'd describe the project.

What’s it all about?

Traditionally, building the workflow for the creation, management, and distribution of media is problematic. It involves having to integrate multiple technologies and providers, some of which may be incompatible. In addition, it can require a huge investment in infrastructure, which may not always be fully utilized. These issues can result in a non-standardized workflow that is not easily scaled, and that requires coordination at different stages of the workflow.

Azure Media Services allow you to build scalable, cost effective, end-to-end media distribution solutions that can upload, encode, package, and stream media to Windows, iOS, Android, Adobe Flash, and other devices and platforms.

The guide describes how to design and build a video-on-demand application that uses Media Services. It provides an end to end walkthrough of a Windows Store application that communicates with Media Services through a web service, and explains key design and implementation choices. In addition to the Windows Store application, client applications are also available for Windows Phone, Web, iOS, and Android.

The guide describes:

• The standard workflow of a media solution that uses Media Services.
• How to use Media Services metadata in a content management system.
• How to securely upload, process, and deliver media.
• How to use Windows Azure Storage Queues to control the encoding process.
• How to deliver video-on-demand in multiple adaptive streaming formats in order to support client applications on different platforms and devices.
• How to scale media encoding and delivery.

In addition to describing the client applications, their integration with Azure Media Services, and the decisions made during the design and implementation, the guide discusses related factors, such as the design patterns used, and the ways that the application could be extended or modified for other scenarios.

The result is that, after reading the guide, you will be familiar with the basic principles of Media Services, and you will be able to design and implement client applications that use Media Services to provide scalable, cost effective, end-to-end media distribution solutions.

What’s in the reference implementation?

The following diagram shows a high-level overview of the reference implementation.

Client applications communicate with the video-on-demand service through a REST web interface. This interface allows applications to retrieve, upload, and publish videos. When a video is uploaded for processing by Media Services it is stored in Microsoft Azure Storage, with the video details being stored in the CMS. It's then encoded to a set of adaptive bitrate MP4s, which can be converted by dynamic packaging to smooth streaming, HLS, or MPEG-DASH, on demand. For more information about dynamic packaging see Azure Media Services dynamic packaging.

When a video is consumed by applications, its URL is retrieved from the CMS and returned to the application. The application then requests the URL content from the Media Services Origin Service, which processes the outbound stream from storage to client app. For more information about the Origin Service, see Azure Media Services Origin Service.

The solution comprises three main components:

• The user facing client applications, implemented for the Windows Runtime, Windows Phone, Web, iOS, and Android. These applications allow users to browse videos, and control the playback of videos. In addition the applications allow users to capture and upload new videos for encoding by Media Services, which can then be consumed from the CMS once encoding has completed.
• The business logic, implemented as a web service. The Contoso web service exposes the data and operations that it supports through a REST interface. Separating the business logic in this way decouples it from the client applications, minimizing the impact that any changes to the implementation of the applications will have on this business logic.
• Data storage, provided by a Microsoft Azure SQL database, and by Microsoft Azure Storage. The CMS, which stores details of videos and encoding jobs, is implemented as a Microsoft Azure SQL database. However, uploaded videos and encoded videos output by Media Services are stored in Microsoft Azure Storage.

Who is it intended for?

The guide and reference implementation is intended for architects, developers, and information technology professionals who design, build, or maintain video-on-demand or online video portal applications and services, particularly those that integrate with content management systems.

Where can I get help?

Patterns for Scalable Media Hosting, like many patterns & practices deliverables, has a community site. On the community site you can post questions, provide feedback, connect with other users to share ideas, and find additional content such as extensions and training material. Community members can also help Microsoft plan and test future releases of Prism for the Windows Runtime. For more info see patterns & practices: Microsoft Azure Media Services Guidance.

Advanced Developers Conference 2014

Yesterday I spoke at Advanced Developers Conference 2014 in Munich about Building a patterned Windows Store app that uses AMP. The slides from the talk can be found here.

The code demos I used are:

• Hilo
• PhotoViewer

For further resources see:

• Guidance: Developing an end-to-end Windows Store app using C++ and XAML: Hilo
• Video: Tips for building a Windows Store app using XAML and C++: The Hilo project
• MSDN: C++ AMP
• Book: C++ AMP: Accelerated Massive Parallelism with Microsoft Visual C++

Thanks to all at ADC for giving me the opportunity to present, and to all attendees for listening.