Thursday, 30 April 2020

Connecting to locahost over HTTPS from simulators and emulators (revisited)

Last year, I wrote about how to connect to localhost over HTTPS from iOS simulators and Android emulators. Connecting to localhost web services that run over HTTP is straight forward, however it’s more work when the web service runs over HTTPS. It typically involves:

  • Creating a self-signed developer certificate on your machine.
  • Choosing the network stack to use.
  • Specifying the address of your local machine.
  • Bypassing the local development certificate check.

Connect to local web services from iOS simulators and Android emulators documents how to carry out the above steps. However, I recently discovered that the approach for bypassing the local development certificate check on iOS no longer worked.

At the time of writing the original blog post, the CoreFX implementation of HttpClient was in most Mono profiles, but not in iOS. The recommendation for iOS was to use the managed network stack, and the ServicePointManager API. However. since then iOS support for the CoreFX HttpClient implementation has dropped, which hooks into the NSUrlSession network stack. This ultimately means that now, on both iOS and Android, the approach for bypassing the local development certificate check is identical.

SSL errors can be ignored on iOS and Android for local HTTPS web services by setting the HttpClientHandler.ServerCertificateCustomValidationCallback property to a callback that ignores the result of the certificate security check for the localhost certificate:

public HttpClientHandler GetInsecureHandler()
{
    HttpClientHandler handler = new HttpClientHandler();
    handler.ServerCertificateCustomValidationCallback = (message, cert, chain, errors) =>
    {
        if (cert.Issuer.Equals("CN=localhost"))
            return true;
        return errors == System.Net.Security.SslPolicyErrors.None;
    };
    return handler;
}
This method must be in a class in a platform project, even if the HttpClient object is constructed in a shared project. The HttpClientHandler object returned by the GetInsecureHandler method should then be passed as an argument to the HttpClient constructor:

HttpClient client = new HttpClient(GetInsecureHandler());
The advantage of using this approach is that it hooks into the native network stacks on both iOS and Android (NSUrlSession on iOS, AndroidClientHandler on Android).

For a full sample that demonstrates this approach in Xamarin.Forms, see my GitHub repo.

Wednesday, 8 April 2020

Combining Xamarin.Essentials WebAuthenticator with IdentityModel.OidcClient

Previously I wrote about using the Xamarin.Essentials WebAuthenticator class to initiate an authentication flow with IdentityServer, and wait for a callback/redirect to the redirect URI scheme. Although my blog post didn’t show it, there was an IdentityService class that created the URI for the authorization endpoint, including the code challenge, CSRF token, and other required query parameters. This class also handled exchanging the received authorization code for an access token, which could then be used in API calls.

While this approach works well, it does require a detailed knowledge of the OAuth 2.0 spec for native apps. While that’s certainly something I’d recommend understanding fully, it would also be convenient if there was a library that did the heavy lifting for you.

Enter IdentityModel.OidcClient

IdentityModel.OidcClient is a C#/.NET standard 2.0 OpenID certified library for native mobile and desktop applications. It’s a small, simple, and elegant library that I strongly recommend for communicating with IdentityServer.

The library has two modes:

  1. In manual mode, OidcClient helps you with creating the necessary start URL and state parameters, but you need to coordinate with the browser you want to use. Then, when the browser work is completed, OidcClient can take over to process the response, get the access/refresh tokens, and contact other endpoints.
  2. In automatic mode, you can encapsulate all browser interactions by implementing the IBrowser interface. Then, authentication and token requests become one line of code.

I’ve previously used manual mode for interacting with IdentityServer, when I had to write my own platform code for invoking native browsers. However, now that WebAuthenticator does that for you, it’s much simpler to use automatic mode and wrap WebAuthenticator using the IBrowser interface.

The IBrowser interface is used to model a browser, and specifies that a single method named InvokeAsync requires implementing, which returns a Task<BrowserResult>. The InvokeAsync method should just invoke your browser to initiate the authentication flow, and return the response in a BrowserResult object.

A Browser class that implements the IBrowser interface, which uses WebAuthenticator to provide the browser interactions, is shown below:

public class Browser : IBrowser
{
    public async Task InvokeAsync(BrowserOptions options, CancellationToken cancellationToken = default)
    {
        WebAuthenticatorResult authResult =
                await WebAuthenticator.AuthenticateAsync(new Uri(options.StartUrl), new Uri(Constants.RedirectUri));
        return new BrowserResult()
        {
            Response = ParseAuthenticatorResult(authResult)
        };
    }

    string ParseAuthenticatorResult(WebAuthenticatorResult result)
    {
        string code = result?.Properties["code"];
        string scope = result?.Properties["scope"];
        string state = result?.Properties["state"];
        string sessionState = result?.Properties["session_state"];
        return $"{Constants.RedirectUri}#code={code}&scope={scope}&state={state}&session_state={sessionState}";
    }
}

Here, the InvokeAsync method calls the WebAuthenticator.AuthenticateAsync method, and returns the parsed response in a BrowserResult object.

An authentication flow can then be initiated by a OidcClient object as follows:

OidcClient _oidcClient;
LoginResult _loginResult;

var options = new OidcClientOptions
{
    Authority = Constants.AuthorityUri,
    ClientId = Constants.ClientId,
    Scope = Constants.Scope,
    RedirectUri = Constants.RedirectUri,
    ResponseMode = OidcClientOptions.AuthorizeResponseMode.Redirect,
    Browser = new Browser()
};

_oidcClient = new OidcClient(options);
_loginResult = await _oidcClient.LoginAsync(new LoginRequest());

Here, an OidcClientOptions object is created that defines the precise authentication flow that will be invoked with IdentityServer. The OidcClientOptions object also defines the IBrowser object that's used to provide browser interactions. An OidcClient object is then created, and its LoginAsync method is invoked to initiate the authentication flow. The result of the authentication flow will then be stored in a LoginResult object, which can include different tokens (access, identity, refresh) along with other data.

The access token can then be used in subsequent API calls to IdentityServer:

HttpClient _httpClient;

_httpClient = new HttpClient();
_httpClient.BaseAddress = new Uri(Constants.ApiUri);
_httpClient.DefaultRequestHeaders.Authorization
    = new AuthenticationHeaderValue("Bearer", _loginResult?.AccessToken ?? string.Empty);

HttpResponseMessage response = await _httpClient.GetAsync("test");
string content = await response.Content.ReadAsStringAsync();

if (response.IsSuccessStatusCode)
{
    editor.Text = JArray.Parse(content).ToString();
}
else
{
    editor.Text = response.ReasonPhrase;
}

Here, the access token is retrieved from the LoginResult object and set as the bearer token in the Authorization header of a HttpClient object: The HttpClient object then makes a call to a specific IdentityServer API, using the access token for authorization.

Wrapping up

This blog post has shown how to combine the Xamarin.Essentials WebAuthenticator class with the IdentityModel.OidcClient library to initiate authentication flows, and process the response. The advantage of combining WebAuthenticator with OidcClient is that (1) you don’t have to write any platform code to invoke native browsers, and (2) you don’t need to write any code to generate the correct authentication query parameters. In addition, because OidcClient is OpenID certified, you have peace of mind that it’s correctly implemented the OAuth 2.0 spec for native apps. For more information about the IdentityModel.OidcClient library, see Building mobile/native clients in the IdentityModel docs.

The sample this code comes from can be found on GitHub.

Tuesday, 7 April 2020

Authentication from a Xamarin.Forms app using WebAuthenticator

The OAuth 2.0 for Native Apps spec represents the best practices for OAuth 2.0 authentication flows from mobile apps. These include:
  • Authentication requests should only be made through external user agents, such as the browser. This results in better security, and enables use of the user’s current authentication state, making single sign-on possible. Conversely, this means that authentication requests should never be made through a WebView. WebView controls are unsafe for third parties, as they leave the authorization grant and user’s credentials vulnerable to recording or malicious use. In addition, WebView controls don’t share authentication state, meaning single sign-on isn’t possible.
  • Native apps must request user authorization by creating a URI with the appropriate grant types. The app then redirects the user to this request URI. A redirect URI that the native app can receive and parse must also be supplied.
  • Native apps must use the Proof Key for Code Exchange (PKCE) protocol, to defend against apps on the same device potentially intercepting the authorization code.
  • Native apps should use the authorization code grant flow with PKCE. Conversely, native apps shouldn’t use the implicit grant flow.
  • Cross-Site Request Forgery (CSRF) attacks should be migitated by using the state parameter to link requests and responses.
Ultimately, the OAuth 2.0 authentication flow for native apps becomes:
  1. The native app opens a browser tab with the authorisation request.
  2. The authorisation endpoint receives the authorisation request, authenticates the user, and obtains authorisation.
  3. The authorisation server issues an authoration code to the redirect URI.
  4. The native app receives the authorisation code from the redirect URI.
  5. The native app presents the authorisation code at the token endpoint.
  6. The token endpoint validates the authorization code and issues the requested tokens.
I’d previously written a Xamarin.Forms sample to do this without any additional libraries, and one with the IdentityModel.OidcClient2 library, which is OpenID certified. Both samples consume endpoints on a publically available IdentityServer site, and use platform code to invoke native browsers.

Enter Xamarin.Essentials


Xamarin.Essentials recently introduced a WebAuthenticator class, which is a web navigation API that can be used for authentication with web services. The class contains a single method, AuthenticateAsync, which starts an authentication flow by navigating to a specified URI, and then waits for a callback/redirect to the redirect URI scheme. The whole point of the class is that it avoids having to write platform code to invoke native browsers. For more information, see the WebAuthenticator doc.

I decided to test drive WebAuthenticator by using it to initate an authentication flow with IdentityServer. However, an authentication flow from a mobile app should include a web backend between the mobile app and the authentication provider. This is to avoid including client secrets in your mobile app, which is considered to be insecure. However, I’ve avoided a web backend as middleware here, as I just want to see what’s involved in authenticating with IdentityServer using the WebAuthenticator class.

However, before you can use WebAuthenticator to authenticate with IdentityServer, you first have to perform some platform specific setup.

Platform setup


On Android, you require an IntentFilter to handle your redirect URI. This can be accomplished by subclassing the Xamarin.Essentials WebAuthenticatorCallbackActivity class:
[Activity(NoHistory = true, LaunchMode = LaunchMode.SingleTop)]
[IntentFilter(new[] { Intent.ActionView},
    Categories = new[] { Intent.CategoryDefault, Intent.CategoryBrowsable},
    DataScheme = "io.identitymodel.native",
    DataHost = "callback")]
public class WebAuthenticationCallbackActivity : WebAuthenticatorCallbackActivity
{
}
You’ll also need to overide the OnResume method in your MainActivity class, to call back into Xamarin.Essentials:
protected override void OnResume()
{
    base.OnResume();
    Xamarin.Essentials.Platform.OnResume();
}
On iOS, you’ll need to add your app’s redirect URI to your Info.plist file:
 <key>CFBundleURLTypes</key>
 <array>
  <dict>
   <key>CFBundleURLName</key>
   <string>WebAuthenticatorDemo</string>
   <key>CFBundleTypeRole</key>
   <string>None</string>
   <key>CFBundleURLSchemes</key>
   <array>
    <string>io.identitymodel.native</string>
   </array>
  </dict>
 </array>
You’ll also need to overide the OpenUrl method in your AppDelegate class, to call back into Xamarin.Essentials:
public override bool OpenUrl(UIApplication app, NSUrl url, NSDictionary options)
{
    if (Xamarin.Essentials.Platform.OpenUrl(app, url, options))
        return true;

    return base.OpenUrl(app, url, options);
}
On UWP, you need to declare your redirect URI in your Package.appxmanifest file:
    <Applications>
      <Application Id="App"
        Executable="$targetnametoken$.exe"
        EntryPoint="WebAuthenticatorDemo.UWP.App">
        ...

        <Extensions>
          <uap:Extension Category="windows.protocol">
            <uap:Protocol Name="io.identitymodel.native">
              <uap:DisplayName>WebAuthenticatorDemo</uap:DisplayName>
            </uap:Protocol>
          </uap:Extension>
        </Extensions>
      
      </Application>
    </Applications>

Start an authentication flow with WebAuthenticator


An authentication flow with IdentityServer can be started by creating the start URI, and passing it to the AuthenticateAsync method along with the redirect URI:
string url = identityService.CreateAuthorizationRequest();
var authResult = await WebAuthenticator.AuthenticateAsync(new Uri(url), new Uri(Constants.RedirectUri));
If you aren’t familiar with the process of building an authorization request to IdentityServer, I recommend you look at the IdentityService class in the sample. It involves creating a long query string that contains data that indicates to IdentityServer what kind of authentication flow to kick off.

The AuthenticateAsync method returns a WebAuthenticatorResult object, which contains a number of properties, such as AccessToken, and RefreshToken. More importantly, it has a Properties dictionary that contains all of the query string or URI fragment properties, that can then be accessed by their key. Therefore, the Properties dictionary can be parsed to build the raw response sent by IdentityServer:
string raw = ParseAuthenticatorResult(authResult);
authorizeResponse = new AuthorizeResponse(raw);

string ParseAuthenticatorResult(WebAuthenticatorResult result)
{
    string code = result?.Properties["code"];
    string idToken = result?.IdToken;
    string scope = result?.Properties["scope"];
    string state = result?.Properties["state"];
    string sessionState = result?.Properties["session_state"];
    return $"{Constants.RedirectUri}#code={code}&id_token={idToken}&scope={scope}&state={state}&session_state={sessionState}";
}
The result of a successful authentication flow is that the AuthorizeResponse object contains an authorization code, that can then be exchanged for the requested tokens:
UserToken userToken = await identityService.GetTokenAsync(authorizeResponse.Code);
If the UserToken object contains an access token, it can be used when making API calls to IdentityServer:
var content = await identityService.GetAsync($"{Constants.ApiUri}test", userToken.AccessToken);

Wrapping up


Although there's a lot going on here, in terms of generating the start URI, and processing the response from the redirect URI, using the WebAuthenticator class has avoided having to write platform code to invoke native browsers. This can be a considerable amount of work, depending on the platform, and it can also be confusing if you aren't familiar with the intricacies of each platform.

I've deliberately left out a lot of code from this blog post, which generates the queries to send to IdentityServer. This is because my aim wasn't to cover the intricacies of IdentityServer, but instead how to initiate an authentication flow with WebAuthenticator, and how to process its response. I also said that all of this is simple, but it’s only simple if you have a good understanding of the OAuth 2.0 spec. If you don’t, it can be overwhelming. Therefore, in my next blog post I’ll look at simplifying all of this by using the IdentityModel.OidcClient library.

The sample this code comes from can be found on GitHub.

Friday, 1 November 2019

Bind from a ControlTemplate to a ViewModel with Xamarin.Forms

The best new feature in Xamarin.Forms 4.3 is relative bindings. Relative bindings provide the ability to set the binding source relative to the position of the binding target, and are created with the RelativeSource markup extension, and set as the Source property of a binding expression. For more informatoin about relative bindings, see Xamarin.Forms Relative Bindings.

Relative bindings support a number of modes, including binding to self, binding to an ancestor, and binding from within a ControlTemplate to the templated parent (the runtime object instance to which the template is applied). They also support binding to a view model from within a ControlTemplate, even when the ControlTemplate binds to the templated parent. This makes it possible to support scenarios such as a ControlTemplate containing a Button that binds to a view model ICommand, while other controls in the ControlTemplate bind to the templated parent. This blog post will look at doing this.

The sample this code comes from can be found on GitHub.

Implementation

To demonstrate this scenario, I have a PeopleViewModel class that defines an ObservableCollection named People, and an ICommand named DeletePersonCommand:

    public class PeopleViewModel
    {
        public ObservableCollection People { get; set; }

        public ICommand DeletePersonCommand { get; private set; }

        public PeopleViewModel()
        {
            DeletePersonCommand = new Command((name) =>
            {
                People.Remove(People.FirstOrDefault(p => p.Name.Equals(name)));
            });

            People = new ObservableCollection
            {
                new Person
                {
                    Name = "John Doe",
                    Description = "Lorem ipsum dolor sit amet, consectetur adipiscing elit..."
                },
                new Person
                {
                    Name = "Jane Doe",
                    Description = "Phasellus eu convallis mi. In tempus augue eu dignissim fermentum..."
                },
                new Person
                {
                    Name = "Xamarin Monkey",
                    Description = "Aliquam sagittis, odio lacinia fermentum dictum, mi erat scelerisque..."
                }
            };
        }
    }

There’s also a ContentPage whose BindingContext is set to a PeopleViewModel instance. The ContentPage contains a StackLayout which uses a bindable layout to bind to the People collection:

<ContentPage ...>
    <ContentPage.BindingContext>
        <local:PeopleViewModel />
    </ContentPage.BindingContext>

    <StackLayout Margin="10,35,10,10"
                 BindableLayout.ItemsSource="{Binding People}"
                 BindableLayout.ItemTemplate="{StaticResource PersonTemplate}" />

</ContentPage>

The ItemTemplate of the bindable layout is set to the PersonTemplate resource:

        <DataTemplate x:Key="PersonTemplate">
            <local:CardView BorderColor="DarkGray"
                            CardName="{Binding Name}"
                            CardDescription="{Binding Description}"
                            ControlTemplate="{StaticResource CardViewControlTemplate}" />
        </DataTemplate>

This DataTemplate specifies that each item in the People collection will be displayed using a CardView object that simply defines CardName, CardDescription, BorderColor, and CardColor bindable properties. The appearance of each CardView object is defined using a ControlTemplate named CardViewControlTemplate:

        <ControlTemplate x:Key="CardViewControlTemplate">
            <Frame BindingContext="{Binding Source={RelativeSource TemplatedParent}}"
                   BackgroundColor="{Binding CardColor}"
                   BorderColor="{Binding BorderColor}"
                   CornerRadius="5"
                   HasShadow="True"
                   Padding="8"
                   HorizontalOptions="Center"
                   VerticalOptions="Center">
                <Grid>
                    <Grid.RowDefinitions>
                        <RowDefinition Height="75" />
                        <RowDefinition Height="4" />
                        <RowDefinition Height="Auto" />
                    </Grid.RowDefinitions>
                    <Label Text="{Binding CardName}"
                           FontAttributes="Bold"
                           FontSize="Large"
                           VerticalTextAlignment="Center"
                           HorizontalTextAlignment="Start" />
                    <BoxView Grid.Row="1"
                             BackgroundColor="{Binding BorderColor}"
                             HeightRequest="2"
                             HorizontalOptions="Fill" />
                    <Label Grid.Row="2"
                           Text="{Binding CardDescription}"
                           VerticalTextAlignment="Start"
                           VerticalOptions="Fill"
                           HorizontalOptions="Fill" />
                    <Button Text="Delete"
                            Command="{Binding Source={RelativeSource AncestorType={x:Type local:PeopleViewModel}}, 
                                              Path=DeletePersonCommand}"
                            CommandParameter="{Binding CardName}"
                            HorizontalOptions="End" />
                </Grid>
            </Frame>
        </ControlTemplate>

The root element of the CardViewControlTemplate is a Frame object. whose BindingContext is set to its templated parent (the CardView). Therefore, the Frame object, and all of its children, will resolve their bindings against CardView properties.

However, the Button within the CardViewControlTemplate binds to both its templated parent (the CardView), and to the ICommand in the PeopleViewModel instance. How is this possible? It’s possible because the Button.Command property redefines its binding source to be the binding context of an ancestor whose binding context type is PeopleViewModel. Let’s delve into this a little more.

The RelativeSource markup extension has a Mode property that can be set to one of the values of the RelativeBindingSourceMode enumeration: Self, FindAncestor, FindAncestorBindingContext, and TemplatedParent. The Mode property is the ContentProperty of the RelativeSourceExtension class, and so explicitly setting it using Mode= can be eliminated. In addition, the RelativeSource markup extension has a AncestorType property. Setting the AncestorType property to a type that derives from Element (any Xamarin.Forms control, or ContentView) will set the Mode property to FindAncestor. Similarly, setting the AncestorType property to a type that doesn’t derive from Element will set the Mode property to FindAncestorBindingContext.

Therefore, the relative binding expression Command=”{Binding Source={RelativeSource AncestorType={x:Type local:PeopleViewModel}}, Path=DeletePersonCommand}” sets the Mode property to FindAncestorBindingContext, because the type specified in the AncestorType property doesn’t derive from Element. The Source property is set the BindingContext property of the ancestor whose binding context is of type PeopleViewModel, which in this case is the StackLayout. The Path part of the expression can then resolve the DeletePersonCommand property. However, the Button.CommandParameter property doesn’t alter its binding source, instead inheriting it from its parent in the ControlTemplate. Therefore, this property binds to the CardName property of the CardView. The overall effect of the Button bindings is that when the Button is clicked, the DeletePersonCommand in the PeopleViewModel class is executed, with the value of the CardName property being passed to the ICommand.

Summary

The overall effect of this code is that StackLayout uses a bindable layout to display a collection of CardView objects:

The appearance of each CardView object is defined by a ControlTemplate, whose controls bind to properties on its templated parent (the CardView). However, the Button in the ControlTemplate redefines its binding source to be an ICommand in a view model. When clicked, the Button removes the specified CardView from the bindable layout:

The sample this code comes from can be found on GitHub.

Thursday, 17 October 2019

Xamarin: Connecting to localhost over HTTPS from simulators and emulators

Most mobile apps consume web services. During the development phase, it’s common to deploy a web service locally and consume it from a mobile app running in a simulator or emulator.

Consuming localhost web services that run over HTTP is straight forward enough. However, it’s more work when the web service runs over HTTPS. This involves:
  • Creating a self-signed developer certificate on your machine.
  • Choosing the network stack to use.
  • Specifying the address of your local machine.
  • Bypassing the local development certificate check.
This process is documented at Connect to Local Web Services from iOS Simulators and Android Emulators.

On iOS and Android, attempting to invoke a local secure web service from an app running in the iOS simulator or Android emulator results in an exception, even when the managed network stack is used. This is because the local HTTPS development certificate is self-signed, and self-signed certificates aren’t trusted by iOS or Android.

There are a number of approaches that can be used to work around this issue, but the typical approach was to use the managed HttpClient implementation in your DEBUG builds, and then set a callback for the System.Net.ServicePointManager.ServerCerticateValidationCallback that ignores the result of the localhost certificate check. This approach worked for both iOS and Android. However, in the last month it stopped working on Android and has left people wondering why.

In the last month, the new CoreFX implementation of HttpClient was dropped into the following Mono profiles:
  • Desktop Mono on Linux and OS X.
  • Web Assembly
  • Android
This implementation does not include the ServicePointManager API, because it’s not part of .NET Core. Instead, it includes the HttpClientHandler.ServerCertificateCustomValidationCallback property (API doc). Therefore, currently, the process for ignoring SSL certificate errors on Android has diverged from iOS.

SSL errors can be ignored on Android for local secure web services by setting the ServerCertificateCustomValidationCallback property to a callback that ignores the result of the certificate security check for the localhost certificate:
        public HttpClientHandler GetInsecureHandler()
        {
            var handler = new HttpClientHandler();
            handler.ServerCertificateCustomValidationCallback = (message, cert, chain, errors) =>
            {
                if (cert.Issuer.Equals("CN=localhost"))
                    return true;
                return errors == System.Net.Security.SslPolicyErrors.None;
            };
            return handler;
        }
This HttpClientHandler object returned by the GetInsecureHandler method should be passed as an argument to the HttpClient constructor. The advantage of using this new approach is that it hooks into the AndroidClientHandler native network stack, which is the recommended network stack on Android. Therefore, it’s no longer necessary to use the managed network stack on Android during development.

On iOS, it’s still recommended to use the managed network stack during development, with the old ServicePointManager API. However, iOS support for the CoreFX HttpClient implementation is in the works, and will hook into the NSUrlSession network stack. Once it’s released, the same approach to bypassing localhost certificate checks can be used on iOS and Android.

For a full sample that demonstrates this approach in Xamarin.Forms, see my GitHub repo.

Friday, 23 August 2019

What’s new in CollectionView in Xamarin.Forms 4.2?

Xamarin.Forms 4.2 was released this week, and includes a number of updates to CollectionView. The main updates are outlined below.

Data

CollectionView now supports loading data incrementally as users scroll through items. This enables scenarios such as asynchronously loading a page of data from a web service, as the user scrolls. In addition, the point at which more data is loaded is configurable so that users don’t see blank space, or are stopped from scrolling. For more information, see Load data incrementally.

Layout

It’s no longer necessary to set the layout orientation of a CollectionView, that uses the ListItemsLayout, with the x:Arguments syntax. Instead, the Orientation property can now be set directly on a ListItemsLayout in XAML. For more information, see Horizontal list.

In addition, CollectionView now supports presenting a header and footer that scroll with the items in the list. The header and footer can be strings, views, or DataTemplate objects. For more information, see Headers and footers.

Scrolling

CollectionView now defines a Scrolled event which is fired to indicate that scrolling occured. For more informatoin, see Detect scrolling.

CollectionView also now includes HorizontalScrollBarVisibility and VerticalScrollBarVisibility properties, which represents whether the horizontal or vertical scroll bar is visible. For more informatoin, see Scroll bar visibility.

In addition, CollectionView defines a ItemsUpdatingScrollMode property which represents the scrolling behaviour of the CollectionView when new items are added to it. This allows, for example, the last item to remain in view when new items are added. For more information, see Control scroll position when new items are added.

Grouping

CollectionView now supports displaying grouped data. However, this functionality is currently only available on iOS, and will be added to Android in due course. For more information, see Grouping.

Monday, 29 July 2019

Frequency filtering with Xamarin.Forms and SkiaSharp

Previously, I wrote about performing convolution in SkiaSharp. I used the CreateMatrixConvolution method from the SKImageFilter class to convolve kernels with the source image. This method allows you to specify kernels of arbitrary size, allows you to specify how the edge pixels in the image are handled, and is lightning fast. I was particularly impressed with the execution speed of the method when considering the size of the source image (4032x3024), and the amount of processing that’s performed when convolving an image with a kernel, particularly for larger kernels.

However, convolution is still considered relatively basic in image processing terms, despite the amount of processing performed. Therefore, I decided to move things up a gear and see what the execution speed would be like when performing frequency filtering, which substantially increases the amount of processing performed on an image.

In this blog post I’ll discuss implementing frequency filtering in SkiaSharp. The sample this code comes from can be found on GitHub.

Implementing frequency filtering

As any computer science undergrad can tell you, a Fourier transform take a signal from the time domain and transforms it into the frequency domain. What this means is it takes a signal, and decomposes it into its constituent frequencies. The world of signal processing is full of applications for the Fourier transform, with one of the stanard applications being musical instrument tuners, which compare the frequency of sound input to the known frequencies of specific notes. Fourier transforms can be performed using analog electronics, digital electronics, and even at the speed of light using optics. For more info about Fourier transforms, see Fourier transform.

Another standard application of the Fourier transform is to perform frequency filtering. A signal can be transformed to the frequency domain, where specific frequencies, or frequency ranges, can be removed. This enables a class of filters to be implemented, known as pass through filters: specifically high-pass, low-pass, and band-pass filters. In terms of image processing, a low-pass filter attentuates high frequencies and retains low frequencies unchanged, with the result being similar to that of a smoothing filter. A high-pass filter attentuates low frequencies, and retains high frequencies unchanged, with the results being similar to that of edge detection. A band-pass filter attentuates very low and very high frequencies, but retains a middle band of frequencies, and is often used to enhance edges while reducing noise.

In terms of image processing, the Fourier transform actually transforms an image from the spatial domain to the frequency domain, where frequency filtering can then be performed. The process of performing frequency filtering on an image is:

  • Fourier transform the image.
  • Filter the image frequencies in the frequency domain.
  • Inverse Fourier transform the frequency data back into the spatial domain.

In reality, the process of Fourier transforming an image is more complex than the bullet points suggest. The frequency data that results from a Fourier transform is expressed as a complex number (a number that has a real, and imaginary component), whose magnitude represents the amount of that frequency present in the original signal, with the phase offset being the basic sinusoid in that frequency. For more information about complex numbers, see complex numbers.

In addition, as well as handling complex numbers, there are other issues to be dealt with when Fourier transforming an image. Firstly, computing a Fourier transform is too slow to be practical, because it has a complexity of O(N2). However, this issue can be overcome by implementing a Fast Fourier Transform (FFT), which has a complexity of O(N log N). For more information about this algorithm, see Fast Fourier transform. The second issue is that Fourier transforms only operate on image dimensions that are a power of 2 (512x512, 1024x1024 etc.). While there are techniques to allow arbitrarily dimensoined images to be Fourier transformed, they are beyond the scope of this blog post. Therefore, any images processed by the algorithm must first be manually resized. The final issue is which image data do you Fourier transform? The answer is to Fourier transform a greyscale representation of the image, as it’s the intensity data that caries more information than colour channels.

For clarity, restating all this leads to the following assumptions for my implementation:

  • The input image for the Fourier transform will be a 32-bit image, with RGBA channels.
  • The input image dimensions must be a power of 2. This is due to the FFT algorithm (Radix-2 FFT) being used.
  • The input image will first be converted to a greyscale representation.
  • The greyscale representation will then be converted to a complex number representation.
  • The complex number data will undergo a 2D FFT.
  • Filtering will then be performed in the frequency domain.
  • The filtered complex data will be inverse FFT’d.
  • The filtered complex data will be converted back from complex numbers to pixel data for display.
  • The output image will be in greyscale, but still stored as a RGBA image.

Implementation

It takes more code to implement a 2D FFT than can be covered in a blog post, particularly as there are complexities I haven’t outlined here. Rather than show every last bit of code, I’ll just state that there’s a Complex type that represents a complex number, and a ComplexImage type that represents an image as a 2D array of complex numbers. The full source code can be found on GitHub.

The following code example shows a high level overview of how the frequency filtering process is implemented:

        public static unsafe SKPixmap FrequencyFilter(this SKImage image, int min, int max)
        {
            ComplexImage complexImage = image.ToComplexImage();

            complexImage.FastFourierTransform();
            FrequencyFilter filter = new FrequencyFilter(new FrequencyRange(min, max));
            filter.Apply(complexImage);
            complexImage.ReverseFastFourierTransform();

            SKPixmap pixmap = complexImage.ToSKPixmap(image);
            return pixmap;
        }

The FrequencyFilter method converts the image to a ComplexImage object, which then undergoes a 2D FFT. A FrequencyFilter object is then created, based on the values of the Slider objects displayed on the UI.The filter is applied to the ComplexImage object, and then inverse FFT’d, before being converted back to pixel data.

A Fourier transform produces a complex number valued output image that can be displayed as two images, for the real and imaginary coefficients, or as their magnitude and phase. In image processing, it’s usually the magnitude of the Fourier transform that’s displayed, as it contains most of the information about the geometric structure of the source image. However, to inverse transform the data after processing in the frequency domain, both the magnitude and phase of the Fourier data is required, and so must be preserved.

It’s possible to view the Fourier transformed image (known as a frequency spectra) by commenting out a couple of lines in the FrequencyFilter method shown above. I mentioned earlier that are additional complexities when implementing a FFT, and one of them is that the dynamic range of Fourier coefficients is too large to be displayed in an image, and the resulting image would appear all black. However, if a logarithmic transformation is applied to the coefficients (which the source code does), the Fourier transformed image can be displayed:

The image is shifted so that Frequency(0,0) is displayed at the center of the image. The further away from the center a point in the image is, the higher is its corresponding frequency. Therefore, this image tells us that the image largely consists of low frequencies. In addition, it’s a fact that low frequencies contain more image information than higher frequencies (which is taken advantage of by image compression algorithms). The spectra also tells us that there’s one dominating direction in the image, which passes vertically through the center. This originates from the many vertical lines present in the source image.

Frequency filtering is performed by the Apply method in the FrequencyFilter class:

        public void Apply(ComplexImage complexImage)
        {
            if (!complexImage.IsFourierTransformed)
            {
                throw new ArgumentException("The source complex image should be Fourier transformed.");
            }

            int width = complexImage.Width;
            int height = complexImage.Height;
            int halfWidth = width >> 1;
            int halfHeight = height >> 1;
            int min = frequencyRange.Min;
            int max = frequencyRange.Max;

            Complex[,] data = complexImage.Data;
            for (int i = 0; i < height; i++)
            {
                int y = i - halfHeight;
                for (int j = 0; j < width; j++)
                {
                    int x = j - halfWidth;
                    int d = (int)Math.Sqrt(x * x + y * y);

                    if ((d > max) || (d < min))
                    {
                        data[i, j].Re = 0;
                        data[i, j].Im = 0;
                    }
                }
            }
        }

This method iterates over the complex image data, and zeros the real and imaginery values that lie outside the frequency range specified by the min and max values. Conversely, frequency data within the min and max values is passed through. This method therefore implements a band-pass filter, which can be configured to operate at any frequency range.

It should therefore follow from this, that if a frequency filter with a min value of 0 and a max value of 1024 is applied, the resulting inverse transformed frequency filtered image should be a perfect greyscale representation of the original source image. The following screenshot shows this:

Furthermore, because the earlier frequency spectra shows that the image is largely comprised of low frequency data, a frequency filter with a min value of 0 and a max value of 128 still results (after inverse FFT) in a perfect greyscale representation of the original source image. The following screenshot shows this:

However, a frequency filter with a min value of 10 and a max value of 128 yields the following image:

In this image, because some of the low frequency data has been removed, only sharp changes in intensity values are being preserved, with the resulting image beginning to look as if it’s been edge detected. Similarly, a frequency filter with a min value of 20 and a max value of 128 furthers this effect:

Again, the output is now looking even more like the output of an edge detector.

While this example is of no immediate practical use, it hopefully shows the potential of what can be achieved with frequency filtering. One of the main uses of frequency filtering in image processing is to remove noise. If the frequency range of the noise can be identified, which it often can be, that frequency range can be removed, resulting in a denoised image. Another real world application of the Fourier transform in imaging, is producing images of steel rebar quality inside concrete (think steel rebar inside concrete walls, bridges etc.). In this case, the transducer data can be deconvolved (in the frequency domain) with the point spread function of the transducer, to yield images of steel rebar quality inside concrete, from which deterioration can be identified.

Wrapping up

What I set out to address here is, is the combination of Xamarin.Forms and SkiaSharp a viable platform for writing cross-platform imaging apps, when performing substantial image processing? My main criteria for answering this question are:

  1. Can most of the app be written in shared code?
  2. Can imaging algorithms be implemented so that they have a fast execution speed, particularly on Android?

The answer to both questions, is yes. I was happy with the speed of execution of the Fourier transform on both platform, especically when considering the size of the source image (2048x2048), and the sheer amount of processing that’s performed when frequency filtering an image. In addition, there are plenty of opportunities to further optimise my implementation, as my implementation focus was clarity rather than optimisation. In particular, a 2D FFT naturally lends itself to parallelisation.

While consumer imaging apps don’t typically contain operations that Fourier transform image data, it’s a mainstay of scientific imaging. Therefore, it’s safe to say that Xamarin.Forms and SkiaSharp also make a good combination for scientific imaging apps.

The sample this code comes from can be found on GitHub.