Monday, 28 July 2014

Encoding and processing media in Azure Media Services

Previously I’ve summarised the media upload process from client apps into a video on-demand service that uses Azure Media Services.

In this blog post I’ll summarise how to incorporate Media Services’ encoding and processing functionality into an app. To do this you create processing jobs that enable you to schedule and automate the encoding and processing of assets.

Encoding and processing media

Media services provides a number of media processors that enable media to be processed. Media processors handle a specific processing task, such as encoding, format conversion, encrypting, or decrypting content. Encoding video is the most common Media Services processing operation, and is performed by the Azure Media Encoder. The Media Encoder is configured using encoder preset strings, with each preset specifying a group of settings required for the encoder. For a list of all the presets see Azure Media Encoder Presets.

Media Services supports progressive download of video and streaming. When encoding for progressive download you encode to a single bitrate. To be able to stream content it must first be converted into a streaming format. There are two types of streaming offered by Media Services:

  • Single bitrate streaming
  • Adaptive bitrate streaming

With single bitrate streaming a video is encoded to a single bitrate stream and divided into chunks. The stream is delivered to the client one chunk at a time. The chunk is displayed and the client then requests the next chunk. When encoding for adaptive bitrate streaming you encode to an MP4 bitrate set that creates a number of different bitrate streams. These streams are also broken into chunks. However, adaptive bitrate technologies allow the client to determine network conditions and select from among several bitrates. When network conditions degrade, the client can select a lower bitrate allowing the video to continue to play at a lower quality. Once network conditions improve the client can switch back to a higher bitrate with improved quality.

Media Services supports three adaptive bitrate streaming technologies:

  1. Smooth streaming, created by Microsoft
  2. HTTP Live Streaming (HLS), created by Apple
  3. MPEG-DASH, an ISO standard
Accessing media processors

Processing jobs involve calling a specific media process to process the job. Media Services supports the following media processors:

Media processor Description
Azure Media Encoder Allows you to run encoding tasks using the Media Encoder,
Azure Media Packager Allows you to convert media assets from MP4 to Smooth Streaming format, and Smooth Streaming assets to HLS format..
Azure Media Encryptor Allows you to encrypt media assets using PlayReady protection.
Storage Decryption Allows you to decrypt media assets that were encrypted using storage encryption.

To use a specific media processor you should pass the name of the processor into the GetLatestMediaProcessorByName method.

private IMediaProcessor GetLatestMediaProcessorByName(string mediaProcessorName)
{
    var processor = this.context.MediaProcessors.Where(p => p.Name == mediaProcessorName)
        .ToList().OrderBy(p => new Version(p.Version)).LastOrDefault();
 
    if (processor == null)
    {
        throw new ArgumentException(string.Format("Unknown media processor: {0}", mediaProcessorName));
    }
                
    return processor;
}

This method retrieves the specified media processor and returns an instance of it. It can be invoked as follows:

IMediaProcessor mediaProcessor = this.GetLatestMediaProcessorByName(MediaProcessorNames.WindowsAzureMediaEncoder);
Creating encoding jobs

After media has been uploaded into Media Services it can be encoded into one of the formats supported by the Media Services Encoder. The Media Services Encoder supports encoding using the H.264 and VC-1 codecs, and can generate MP4 and Smooth Streaming content. However, MP4 and Smooth Streaming content can be converted to HLS v3 or MPEG-DASH by using dynamic packaging.

Encoding jobs are created and controlled using a Job. Each Job contains metadata about the processing to be performed, and contains one or more Tasks that specify a processing task, its input Assets, output Assets, and a media processor and its settings. Tasks within a Job can be chained together, where the output asset of one task is given as the input asset to the next task. By following this approach one Job can contain all of the processing required for a media presentation.

The following diagram shows a high-level overview of the media encoding process used in the Building an On-Demand Video Service with Microsoft Azure Media Services project.

untitled1

The EncodingService class retrieves the asset details from the CMS database and passes the encoding job to Media Services, where it’s submitted to the Azure Media Encoder. The encoding job and video details are saved to the CMS database while the Media Encoder processes the job, retrieving the input asset from Azure Storage, and writing the output assets to Azure Storage. When encoding is complete Media Services notifies the EncodingService class, which generate locator URLS to the output assets in Azure Storage, and updates the encoding job and video details in the CMS database. For a code walkthrough of this process see Encoding process in the Contoso Azure Media Services web service.

By default, each Media Services account can have one active encoding task at a time. However, you can reserve encoding units that allow you to have multiple encoding tasks running concurrently. For more information see How to Scale a Media Service.

Accessing encoded media

Accessing content in Media Services requires a locator, which combines the URL to the media file with a set of time-based access permissions. There are two types of locators – shared access signature (SAS) locators and on-demand origin locators.

A SAS locator grants access rights to a specific media asset through a URL. By using the URL you are granting users who have the URL access to a specific resource for a period of time, in addition to specifying what operations can be performed on the resource.

On-demand origin locators are used when streaming content to a client app, and are exposed by the Media Services Origin Service which pulls the content from Azure Storage and delivers it to the client. An on-demand origin locator URL will point to a streaming manifest file in asset. For more information about the Origin Service see Origin Service.

Summary

This blog post has summarised how to use the Azure Media Encoder to encode uploaded media for delivery to client apps.To do this you create a media processing job that enables you to schedule and automate the encoding of assets. For more info see Building an On-Demand Video Service with Microsoft Azure Media Services.

In my next blog post I’ll discuss the final step in the Media Services workflow – delivering and consuming media.

Friday, 25 July 2014

Using a custom overlay in a Windows Phone QR code scanning app

Previously I’ve demonstrated how to build a simple Windows Phone app to perform QR code scanning, using the ZXing.Net.Mobile library. This library makes the scanning and decoding of bar codes effortless, leaving you to focus on other user experiences in your app.

In this blog post I’m going to extend the sample app so that it doesn’t use the default UI included with ZXing.Net.Mobile, when QR code scanning. Instead, a custom UI will be created and used during the QR code scanning process. This custom UI is referred to as an overlay.

Implementation

The first step is to define the overlay in the XAML code for the page.

<Grid Name="Overlay" Visibility="Collapsed">
    <Grid Background="Transparent">
        <Grid.RowDefinitions>
            <RowDefinition Height="*" />
            <RowDefinition Height="Auto" />
        </Grid.RowDefinitions>
        <Grid.ColumnDefinitions>
            <ColumnDefinition Width="*" />
            <ColumnDefinition Width="*" />
        </Grid.ColumnDefinitions>
        <Button Background="Black" Grid.Row="1" Grid.Column="0" Name="ButtonCancel">Cancel</Button>
        <Button Background="Black" Grid.Row="1" Grid.Column="1" Name="ButtonTorch">Torch</Button>
    </Grid>
</Grid>

This code defines a custom overlay named Overlay, which contains a Cancel button and a Torch button (for toggling on/off the flash) that will appear at the bottom of the page. The next step is pass the overlay to the ZXing.Net.Mobile library.

private UIElement _overlay = null;
 
private async void MainPage_Loaded(object sender, RoutedEventArgs e)
{
    _scanner = new MobileBarcodeScanner(this.Dispatcher);
 
    if (_overlay == null)
    {
        _overlay = this.Overlay.Children[0];
        this.Overlay.Children.RemoveAt(0);
    }
 
    this.ButtonCancel.Click += (s, e2) =>
        {
            _scanner.Cancel();
        };
    this.ButtonTorch.Click += (s, e2) =>
        {
            _scanner.ToggleTorch();
        };
 
    _scanner.CustomOverlay = _overlay;
    _scanner.UseCustomOverlay = true;
 
    var result = await _scanner.Scan();
    ProcessScanResult(result);
}

After creating an instance of the MobileBarcodeScanner class the Overlay is retrieved from the visual tree and stored in a UIElement instance named _overlay. Then, the Cancel and Torch button Click events are wired up to methods in the MobileBarcodeScanner class, to cancel scanning, and toggle the torch respectively. The CustomOverlay property of the MobileBarcodeScanner instance is then set to _overlay, the UseCustomOverlay property of the MoblileBarcodeScanner instance is set to true to indicate that a custom overlay will be used during the QR scanning process, before the Scan method of the MobileBarcodeScanner instance is invoked. The following screenshot shows the custom overlay being displayed during the QR code scanning process (QR code image courtesy of Google Images):

image

While the overlay shown here is basic, it does show the mechanism used for creating a UI for the QR code scanning process that matches the rest of the UI used in your app.

When a QR code is successfully recognized the decoded result can be passed to a method for processing, in this case the ProcessScanResult method. For an explanation of this method see my previous blog post.

Summary

This blog post has demonstrated how to replace the default UI used by ZXing.Net.Mobile, with a custom UI of your own design. This allows you to create a UI for the QR code scanning process that matches the rest of the UI used in your app.

The sample app can be downloaded here.

Thursday, 24 July 2014

QR code scanning in a Windows Phone app

A recent Windows Phone app that I developed had a requirement to include QR code scanning, and to take an action based on the data contained in the QR code. A QR code is a type of machine readable barcode that contains data that can be extracted from patterns present in both horizontal and vertical components of the image. Initially this seemed like quite a big task, but with the help of a library it became an effortless task.

This blog post demos a simple Windows Phone 8 app for performing QR code scanning using ZXing.Net.Mobile. ZXing.Net.Mobile is a .NET library based on the widely used open source ZXing (Zebra Crossing) barcode library, whose goal is to make scanning barcodes as painless as possible in an app.

Implementation

The first step is to add the ZXing.Net.Mobile nuget package to your app, and import the ZXing.Mobile namespace into your class.

private MobileBarcodeScanner _scanner;
 
private async void MainPage_Loaded(object sender, RoutedEventArgs e)
{
    _scanner = new MobileBarcodeScanner(this.Dispatcher);
    _scanner.UseCustomOverlay = false;
    _scanner.TopText = "Hold camera up to QR code";
    _scanner.BottomText = "Camera will automatically scan QR code\r\n\rPress the 'Back' button to cancel";
 
    var result = await _scanner.Scan();
    ProcessScanResult(result);
}

Scanning a QR code is as simple as creating an instance of the MobileBarcodeScanner class, and invoking it’s Scan method. Optional setup includes specifying text to display at the top and bottom of the page, while scanning a QR code, and a flag that indicates whether you want to use ZXings in-built overlay, or specify your own.

The following screenshot shows the app scanning a QR code (courtesy of Google Images).

image

When a QR code is successfully recognized the decoded result can be passed to a method for processing, in this case the ProcessScanResult method. Note that decoding of the image is all automatically performed by the ZXing.Net.Mobile library.

private void ProcessScanResult(ZXing.Result result)
{
    string message = string.Empty;
 
    message = (result != null && !string.IsNullOrEmpty(result.Text)) ? "Found QR code: " + result.Text : "Scanning cancelled";
 
    this.Dispatcher.InvokeAsync(() =>
        {
            MessageBox.Show(message);
        });
}

This method simply extracts the text from the ZXing.Result instance and displays it in a mesage box, as shown in the following screenshot.

image

Obviously in a real app you may choose to log the decoded QR code result somewhere, or in the case of a URL, open it in a web browser.

Summary

This blog post has demonstrated how to construct a simple app for performing QR code scanning, using the ZXing.Net.Mobile library. This library makes the scanning and decoding of bar codes effortless, leaving you to focus on other user experiences in your app.

The sample app can be downloaded here.

Monday, 23 June 2014

Uploading content into Azure Media Services

Previously I’ve introduced the Content Management System (CMS) that’s used to store video details for the video on-demand service, along with details of video encoding jobs. I’ve also explained how to connect to Azure Media Services.

In this blog post I’ll summarise how client apps can upload video to the video on-demand service. For info about the supported codecs and file container formats see Formats Supported by the Media Services Encoder.

Uploading content

To upload content into Media Services you must first create an asset and add files to it, and then upload the asset. This process is known as ingesting content. The content object in Media Services is an IAsset, which is a collection of metadata about a set of media files. Each IAsset contains one or more IAssetFile objects. The approach adopted here is to create an Asset, upload the content to Media Services, and then generate AssetFiles and associate them with the Asset. In order to create an Asset you must first have a reference to the Media Services server context. For more info see Connecting to Azure Media Services.

The following diagram shows a high-level overview of the media upload process used in the Building an On-Demand Video Service with Microsoft Azure Media Services project.

IC721592

The client apps communicate with the web service through a REST web interface. When a new video is uploaded a new asset is created by Media Services, and the asset is uploaded to Azure Storage before the asset details are published to the CMS. The upload process can be decomposed into the following steps:

  1. Create a new empty Asset.
  2. Create an AccessPolicy instance that defines the permissions and duration of access to the Asset.
  3. Create a Locator instance that will provide access to the Asset.
  4. Upload the file that’s associated with the Asset into blob storage.
  5. Publish the Asset.
    • Save the Asset details to the CMS.
    • Generate an AssetFile for the Asset.
    • Add the Asset to the encoding pipeline.

For a code walkthrough of this process see Upload process in the Contoso Azure Media Services applications.

Media Services also allows you to secure your content from the time it leaves your computer, by specifying an encryption option as a parameter when creating an Asset. For more info see Securing media for upload into Azure Media Services.

Summary

This blog post has summarised the media upload process from client apps into a video on-demand service that uses Azure Media Services. For more info see Building an On-Demand Video Service with Microsoft Azure Media Services.

In my next blog post I’ll discuss the next step in the Media Services workflow – encoding and processing uploaded media.

Friday, 20 June 2014

Connecting to Azure Media Services

Before you can start programming against Azure Media Services you need to create a Media Services account in a new or existing Azure subscription. For more info see How to Create a Media Services Account. Once you’ve setup a Media Services account you’ll have connection values for the account in the form of the account name and account key. These values can be stored in configuration and programmatically retrieved to make connections to Media Services.

To code against Media Services you create a CloudMediaContext instance that represents the server context. Media Services controls its access to its services through an OAuth protocol that requires an Access Control Service (ACS) token that is received from an authorisation server. One of the CloudMediaContext constructor overloads takes a MediaServicesCredentials object as a parameter, which enables the reuse of ACS tokens between multiple contexts. You could use a constructor overload that ignores ACS tokens, thus leaving the Media Services SDK to manage them for you. However, this can lead to unnecessary token requests which can create performance issues on both the client and server. The following code example shows how to reuse ACS tokens between multiple contexts.

public class EncodingService
{
    private static readonly Lazy<MediaServicesCredentials> Credentials =
        new Lazy<MediaServicesCredentials>(() =>
        {
            var credentials = new MediaServicesCredentials(
                CloudConfiguration.GetConfigurationSetting("AccountName"),
                CloudConfiguration.GetConfigurationSetting("AccountKey"));
            
            credentials.RefreshToken();
            return credentials;
        });
    
    public EncodingService()
    {
       this.context = new CloudMediaContext(EncodingService.Credentials.Value);
    }
}

The Credentials object is cached in memory as a static class variable that uses lazy initialisation to defer the creation of the object until it is first used. This object contains an ACS token that can be reused if it hasn’t expired. If it has expired it will automatically be refreshed by the Media Services SDK using the credentials given to the MediaServicesCredentials constructor. The cached object is then used by the EncodingService constructor. It’s particularly important to cache your Media Services credentials in a multi-tenant application, otherwise performance issues could occur as a result of thread contention issues.

When the CloudMediaContext instance is created the Credentials object will be created. Using lazy initialisation to do this reduces the likelihood of the MediaServicesCredentials object having to refresh its ACS token due to expiration. For more info about lazy initialisation see Lazy Initialization.

Monday, 2 June 2014

Developing the content management system for a video on-demand service

Previously I’ve introduced the architecture of the video on-demand service, and the Windows Phone app that consumes the service through a REST interface.

In this blog post I’ll discuss the Content Management System (CMS) created for the Building an On-Demand Video Service with Microsoft Azure Media Services project.

Designing the CMS

A video CMS enables you to upload, store, process, and publish media, and generally store data in a database that allows for metadata tagging and searching.

The requirements for our CMS were:

  • Ability to store details of videos that can be consumed by client apps.
  • Ability to store video metadata.
  • Ability to store a thumbnail image for each video.
  • Ability to store details of video encoding jobs.

We decided to store this information in a series of database tables that were optimized for many of the common queries performed by our client apps. The following diagram shows the database table structure.

The table structure was implemented as an Azure SQL relational database as they offer elasticity that enables a system to quickly and easily scale as the number of requests and volume of work increases. An additional advantage is that Azure SQL databases maintain multiple copies of the database on different servers. Therefore, if the primary server fails, all requests are transparently switched to another server.

Accessing the CMS

A relational database stores data as a collection of tables. However, the Windows Phone app processes data in the form of entity objects. The data for an entity object might be constructed from one or more rows in one or more tables. In the Windows Phone app the business logic that manipulates objects is independent of the format of the data for the object in the database. This offers the advantages that you can modify and optimize the database table structure without affecting the code in the Windows Phone app, and vice versa.

This approach requires the use of an object-relational mapping (ORM) layer. The purpose of the ORM is to act as an abstraction of the underlying database. The Windows Phone app creates and uses objects, and the ORM exposes methods that can take those objects and use them to generate relational CRUD operations, which it then sends to the database server. Tabular data is then returned from the database and converted into a set of objects by the ORM.

We used the Entity Framework as the ORM, and used the Fluent API to decouple the classes in the object model from the Entity Framework. In the Entity Framework the database is interacted with through a context object. This object provides the connection to the database and implements the logic performing CRUD operations on the data in the database. The context object also performs the mapping between the object model of the Windows Phone app and the tables defined in the database.

The Windows Phone app sends REST requests to the video on-demand service, which validates the requests and converts them into the corresponding CRUD operations against the CMS. All incoming REST requests are routed to a controller based on the URL that the Windows Phone app specifies. The controllers indirectly use the Entity Framework to connect to the CMS database and perform CRUD operations. We implemented the Repository pattern in order to minimize the dependencies that the controllers have on the Entity Framework.

The purpose of the Repository pattern is to act as an intermediary between the ORM layer and the data mapping layer that provides the objects for the controller classes. In the video on-demand service, each repository classes provides a set of APIs that enable a service class to retrieve a database-neutral object from the repository, modify it, and store it back in the repository. The repository class has the responsibility for converting all the requests made by a service class into commands that it can pass to the Entity Framework. As well as removing any database-specific dependencies from the business logic of the controller and service classes, this approach provides flexibility. If we chose to switch to a different data store we could provide alternative implementations of the repository classes that expose the same APIs to the service classes.

For more information about the video on-demand service and its use of the Repository pattern see Appendix A – The Contoso Web Service.

Summary

This blog post has discussed why we chose to store the CMS database in the cloud as an Azure SQL database. It has explained how the web service connects to the database by using the Entity Framework, and how the Repository pattern is used to abstract the details of the Entity Framework from the business logic of the video on-demand service. For more information see Building an On-Demand Video Service with Microsoft Azure Media Services.

In my next blog post I’ll discuss how the Windows Phone app uploads video to the video on-demand service.