Fault Handling in Xamarin.Forms: Combining the Retry and Circuit Breaker Patterns

Previously, I’ve written about using the retry pattern to perform transient fault handling in Xamarin.Forms with Polly, and using the circuit breaker pattern to handle variable length faults with Polly. The retry pattern enables an application to retry an operation in the expectation that it will eventually succeed, while the circuit breaker pattern prevents an application from performing an operation that is likely to fail.

These patterns can both increase the stability and resiliency of an application. So how do you choose which to implement in your application? Do you choose to only handle transient faults, at the expense of ignoring variable length faults? Or do you choose to ignore transient faults, and instead handle variable length faults?

Ideally you should handle both faults. An application can combine the patterns by using the retry pattern to invoke an operation through a circuit breaker. The only caveat is that the retry logic should be sensitive to any exceptions returned by the circuit breaker, and it should abandon retry attempts if the circuit breaker indicates that a fault is not transient.

This blog post will discuss using Polly’s PolicyWrap class to combine the retry and circuit breaker patterns.

Implementation

The sample application, which can be found on GitHub, is similar to the sample application from my previous blog posts in this series, with the implementation of the retry and circuit breaker patterns being provided by Polly.

Initialization

The App class in the sample application initializes the classes that are responsible for communicating with the REST service:

TodoManager = new TodoItemManager(
    new RestService(
        new ResilientRequestProvider()));

The RestService class provides data to the TodoItemManager class, with the RestService class making REST calls using the ResilientRequestProvider class, which uses Polly to combine the retry and circuit breaker patterns.

ResilientRequestProvider

The following code example shows the GetAsync method from the ResilientRequestProvider class, which makes GET requests to a specified URI:

async Task<HttpResponseMessage> HttpInvoker(Func<Task<HttpResponseMessage>> operation)
{
    return await policyWrap.ExecuteAsync(operation);
}

public async Task<TResult> GetAsync<TResult>(string uri)
{
    string serialized = null;
    var httpResponse = await HttpInvoker(async () =>
    {
        var response = await client.GetAsync(uri);
        response.EnsureSuccessStatusCode();
        serialized = await response.Content.ReadAsStringAsync();
        return response;
    });
    return JsonConvert.DeserializeObject<TResult>(serialized);
}

The lambda expression in the GetAsync method is passed to the HttpInvoker method, which in turn passes it to the ExecuteAsync method of the PolicyWrap instance. The PolicyWrap type is a Polly type, which provides a way to combine resilience strategies. Therefore, the code in the lambda expression is what will be executed by the policy that wraps the retry and circuit breaker policies.

The following code example shows how to combine Polly’s retry and circuit breaker policies:

var retryPolicy = Policy
    // Don't retry if circuit breaker has broken the circuit
    .Handle<Exception>(e => !(e is BrokenCircuitException))
    .WaitAndRetryAsync(3,
                       retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)),
                       (exception, delay, retryCount, context) =>
                       {
                           Debug.WriteLine($"Retry {retryCount} after {delay.Seconds} seconds delay 
                                             due to {exception.Message}");
                       });

var circuitBreakerPolicy = Policy
    .Handle<Exception>()
    .CircuitBreakerAsync(4,
                         TimeSpan.FromSeconds(5),
                         // onBreak
                         (exception, delay) => 
                             Debug.WriteLine($"Breaking the circuit for {delay.Seconds} seconds 
                                               due to {exception.Message}"),
                         // onReset
                         () => Debug.WriteLine($"Call ok - closing the circuit again."),
                         // onHalfOpen
                         () => Debug.WriteLine($"Circuit is half-open. The next call is a trial."));

policyWrap = Policy.WrapAsync(retryPolicy, circuitBreakerPolicy);

The wrapped policies are executed by the PolicyWrap.ExecuteAsync method from the outermost (left) to the innermost (right), with exceptions bubbling back outwards through the layers. Therefore, retryPolicy is the outermost policy and it places its call through the circuitBreakerPolicy. The circuitBreakerPolicy is the innermost policy, and unless the circuit is open, it executes the passed delegate. The passed delegate either succeeds, returns handled results, or throws an exception. The circuit breaker will then update the circuit state for that outcome, and pass it back to the retryPolicy. The retryPolicy will then, provided that the circuit breaker hasn’t thrown a BrokenCircuitException, (for a fault) wait and retry the operation, or fail when retries are exhausted.

For more information about using Polly’s retry policy, see Transient Fault Handling in Xamarin.Forms using Polly. For more information about using Polly’s circuit breaker policy, see Fault Handling in Xamarin.Forms: Circuit Breaker using Polly.

Running the Sample Application

The sample application, which can be found on GitHub, connects to a read-only REST service hosted by Xamarin, and it’s most likely that when running the sample the GET operation will succeed on first attempt. To observe the combined retry and circuit breaker patterns in action, change the RestUrl property in the Constants class to an address that doesn’t exist – this can be accomplished by adding a random character to the end of the existing string. Then run the application and observe the output window in Visual Studio. You should see something like:

Retry 1 after 2 seconds delay due to 404 (Not Found)
Retry 2 after 4 seconds delay due to 404 (Not Found)
Retry 3 after 8 seconds delay due to 404 (Not Found)
Breaking the circuit for 5 seconds due to 404 (Not Found)

This shows the GET operation being retried 3 times, after an exponentially increasing delay. After the third retry, the circuit is broken for the specified amount of time. Note that the values used to configure the retry and circuit breaker policies in this example are purely for demonstration purposes.

Summary

Polly is a .NET fault handling library, which includes fluent support for a number of resilience patterns. The PolicyWrap type allows policies to be easily combined, with them being executed from the outermost to the innermost.

The advantage of using Polly over implementing your own resilience patterns is that Polly includes multiple fault handling patterns that can easily be combined for additional resilience when handling faults.

Xamarin.Forms 2.5 and Local Context on Android

Previously I wrote about Xamarin.Forms 2.5 and local context in Android custom renderers. I briefly mentioned that Xamarin.Forms.Forms.Context is now obsolete, and that in a custom renderer you can usually replace references to Xamarin.Forms.Forms.Context with a reference to the view’s context, which is passed into the custom renderer constructor.

So what do you do if your app references Xamarin.Forms.Forms.Context in a class that isn’t a custom renderer? For example, consider the GetVersionNumber method in the VersionHelper class:

using Android.Content;

namespace DependencyServiceAndLocalContext.Droid
{
    public class VersionHelper : IVersionHelper
    {
        public string GetVersionNumber()
        {
            var versionNumber = string.Empty;
            Context context = Xamarin.Forms.Forms.Context;
            if (context != null)
            {
                versionNumber = context.PackageManager.GetPackageInfo(context.PackageName, 0).VersionName;
            }
            return versionNumber;
        }            
    }
}

This code is executed by Xamarin.Forms DependencyService class, and references Xamarin.Forms.Forms.Context to get the local context. The problem is that as of Xamarin.Forms 2.5, it will result in a compiler warning - ‘Forms.Context’ is obsolete: ‘Context is obsolete as of version 2.5. Please use a local context instead.’

The solution is to get the local context without referencing Xamarin.Forms.Forms.Context. How this is achieved depends on whether your app has a single Activity, or multiple activities.

Note: my preferred approach is the technique documented for apps with multiple activities, as it’s also applicable to apps with only a single Activity. Therefore, this is the approach I’ve demonstrated in a sample app that can be found on GitHub.

Single Activity Apps

In a single Activity app the local context can be obtained from the MainActivity class. This can be achieved by adding an Init method to the VersionHelper class:

using Android.Content;

namespace DependencyServiceAndLocalContext.Droid
{
    public class VersionHelper : IVersionHelper
    {
        static Context _context;

        public static void Init(Context context)
        {
            _context = context;
        }

        public string GetVersionNumber()
        {
            var versionNumber = string.Empty;
            if (_context != null)
            {
                versionNumber = _context.PackageManager.GetPackageInfo(
                    _context.PackageName, 0).VersionName;
            }
            return versionNumber;
        }
    }
}

Then in the MainActivity class, call the VersionHelper.Init method, passing in the MainActivity instance as the context:

VersionHelper.Init(this);

Alternatively, create a static property in the MainActivity class, and set it to the MainActivity instance:

public class MainActivity : global::Xamarin.Forms.Platform.Android.FormsAppCompatActivity
{
    internal static MainActivity Instance { get; private set; }

    protected override void OnCreate(Bundle bundle)
    {
        ...
        global::Xamarin.Forms.Forms.Init(this, bundle);
        Instance = this;
        Xamarin.Forms.DependencyService.Register<IVersionHelper, VersionHelper>();
        LoadApplication(new App());
    }
}

Then, in the VersionHelper class, retrieve the local context from the MainActivity class:

using Android.Content;

namespace DependencyServiceAndLocalContext.Droid
{
    public class VersionHelper : IVersionHelper
    {
        public string GetVersionNumber()
        {
            var versionNumber = string.Empty;
            if (MainActivity.Instance != null)
            {
                versionNumber = MainActivity.Instance.PackageManager.GetPackageInfo(
                    MainActivity.Instance.PackageName, 0).VersionName;
            }
            return versionNumber;
        }
    }
}

There are other variations on this theme, but they all amount to the same idea – retrieving the local context from the MainActivity class.

Multiple Activity Apps

If your app uses multiple activities, and hence you need to ensure that the local context is accurate, the approach outlined for single Activity apps can be used and modified, but it soon becomes messy to ensure that classes have references to the correct local context.

A better approach is to create a MainApplication class that implements the IActivityLifecycleCallbacks interface:

using System;
using Android.App;
using Android.Content;
using Android.OS;
using Android.Runtime;

namespace DependencyServiceAndLocalContext.Droid
{
    [Application]
    public partial class MainApplication : Application, Application.IActivityLifecycleCallbacks
    {
        internal static Context CurrentContext { get; private set; }

        public MainApplication(IntPtr handle, JniHandleOwnership transfer) : base(handle, transfer)
        {
        }

        public override void OnCreate()
        {
            base.OnCreate();
            RegisterActivityLifecycleCallbacks(this);
        }

        public override void OnTerminate()
        {
            base.OnTerminate();
            UnregisterActivityLifecycleCallbacks(this);
        }

        public void OnActivityCreated(Activity activity, Bundle savedInstanceState)
        {
            CurrentContext = activity;
        }

        public void OnActivityDestroyed(Activity activity)
        {

        }

        public void OnActivityPaused(Activity activity)
        {

        }

        public void OnActivityResumed(Activity activity)
        {
            CurrentContext = activity;
        }

        public void OnActivitySaveInstanceState(Activity activity, Bundle outState)
        {

        }

        public void OnActivityStarted(Activity activity)
        {
            CurrentContext = activity;
        }

        public void OnActivityStopped(Activity activity)
        {

        }
    }
}

The CurrentContext property is updated with a reference to the current Activity, whenever an Activity is created, started, or resumed. Therefore this property will always have a reference to the current Activity, which can then be used as the local context. Therefore, the GetVersionNumber method in the VersionHelper class becomes:

using Android.Content;

namespace DependencyServiceAndLocalContext.Droid
{
    public class VersionHelper : IVersionHelper
    {
        public string GetVersionNumber()
        {
            var versionNumber = string.Empty;
            if (MainApplication.CurrentContext != null)
            {
                versionNumber = MainApplication.CurrentContext.PackageManager.GetPackageInfo(
                    MainApplication.CurrentContext.PackageName, 0).VersionName;
            }
            return versionNumber;
        }
    }
}

By using this approach, any references to Xamarin.Forms.Forms.Context can simply be replaced with MainApplication.CurrentContext. In addition, while this approach is aimed at apps that contain multiple activities, it has the advantage that it can also be used by apps that contain only a single Activity.

Xamarin.Forms 2.5 and Local Context in Android Custom Renderers

If you’ve updated any of your solutions to use Xamarin.Forms 2.5, you may have noticed compiler warnings that say “Context is obsolete as of version 2.5. Please use a local context instead”. This warning typically occurs on Android custom renderers, and any references to Xamarin.Forms.Forms.Context.

The warning occurs on Android custom renderers because the parameterless constructors have been marked obsolete. This change was made to support native embedding in Android apps that use multiple activities. Instead, the custom renderer base classes now have constructors that take a Android.Content.Context argument. In addition, Xamarin.Forms.Forms.Context has been marked obsolete because it was a global static reference to an Activity, which is not ideal.

Custom renderer implementations in applications can be fixed by adding a constructor that takes a Android.Content.Context argument and passing it to the base class constructor, as shown in the following code example:

[assembly: ExportRenderer(typeof(MyEntry), typeof(MyEntryRenderer))]
namespace CustomRenderer.Android
{
    class MyEntryRenderer : EntryRenderer
    {
        public MyEntryRenderer(Context context) : base(context)
        {
        }

        protected override void OnElementChanged(ElementChangedEventArgs<Entry> e)
        {
            base.OnElementChanged(e);

            if (Control != null)
            {
                Control.SetBackgroundColor(global::Android.Graphics.Color.LightGreen);
            }
        }
    }
}

Technically the constructor isn’t required (unless you’re using native embedding in an Android app that has multiple activities) – the backwards compatibility built into Xamarin.Forms will take care of it. However, adding the constructor will eliminate the compiler warning, and produce some peace of mind.

If your custom renderer accesses Xamarin.Forms.Forms.Context, you should most likely just be using the view’s context, which is passed into the custom renderer constructor.

Fault Handling in Xamarin.Forms: Circuit Breaker using Polly

Previously I wrote about about handling faults that take a variable amount of time to recover from, in Xamarin.Forms, and discussed an implementation of the circuit breaker pattern. The advantage of the implementation was that the circuit breaker pattern was implemented without requiring any library code, for those sensitive to bloating their application package size.

There are, however, fault handling libraries available and the go to library for .NET is Polly, which includes fluent support for the circuit breaker pattern, and much more. In Polly, the various patterns are implemented via fault handling policies, which handle specific exceptions thrown by, or results returned by, the delegates that are executed through the policy.

This blog post will discuss using Polly’s CircuitBreakerPolicy, which implements the circuit breaker pattern.

Implementation

The sample application, which can be found on GitHub, is similar to the sample application from my previous blog post, with the custom implementation of the circuit breaker pattern replaced with Polly’s.

Initialization

The App class in the sample application initializes the classes that are responsible for communicating with the REST service:

TodoManager = new TodoItemManager(
    new RestService(
        new ResilientRequestProvider()));

The RestService class provides data to the TodoItemManager class, with the RestService class making REST calls using the ResilientRequestProvider class, which uses Polly to implement the circuit breaker pattern.

ResilientRequestProvider

The following code example shows the GetAsync method from the ResilientRequestProvider class, which makes GET requests to a specified URI:

async Task<HttpResponseMessage> HttpInvoker(Func<Task<HttpResponseMessage>> operation)
{
    return await circuitBreakerPolicy.ExecuteAsync(operation);
}

public async Task<TResult> GetAsync<TResult>(string uri)
{
    string serialized = null;
    var httpResponse = await HttpInvoker(async () =>
    {
        var response = await client.GetAsync(uri);
        response.EnsureSuccessStatusCode();
        serialized = await response.Content.ReadAsStringAsync();
        return response;
    });
    return JsonConvert.DeserializeObject<TResult>(serialized);
}

The GetAsync method code is identical to my previous blog post. The lambda expression is passed to the HttpInvoker method, which in turn passes it to the ExecuteAsync method of the CircuitBreakerPolicy instance. Therefore, the code in the lambda expression is what will be executed by the circuit breaker, provided it’s in a state that allows that.

There are three steps to using a fault handling policy, including the CircuitBreakerPolicy, in Polly:

1. Specify the exceptions you want the policy to handle.
2. Optionally specify the returned results you want the policy to handle.
3. Specify how the policy should handle any faults.

The following code example shows the first and third steps, used in the sample application, for defining the operation of the CircuitBreakerPolicy instance:

circuitBreakerPolicy = Policy
    .Handle<Exception>()
    .CircuitBreakerAsync(
        1,
        TimeSpan.FromSeconds(5),
        // onBreak
        (exception, delay) => 
        Debug.WriteLine($"Breaking the circuit for {delay.TotalSeconds} due to {exception.Message}"),
        // onReset
        () => 
        Debug.WriteLine($"Call ok - closing the circuit again."),
        // onHalfOpen
        () => 
        Debug.WriteLine($"Circuit is half-open. The next call is a trial."));

The Policy.Handle method is used to specify the exceptions you want the policy to handle. In this example the circuit breaker will trigger after a specified number of consecutive exceptions of the handled type (Exception).

After specifying the exceptions the policy should handle, you must specify how the policy should handle any faults with the CircuitBreakerAsync method. There are multiple overloads for this method, and I used one for which five arguments must be specified:

1. The consecutive number of exceptions that must be thrown to break the circuit. This is set to 1 in the sample application, purely for demonstration purposes.
2. The duration to keep the circuit broken for. This is set to 5s in the sample application, purely for demonstration purposes.
3. An action to be called when the circuit changes to an Open state.
4. An action to be called when the circuit changes to a Closed state.
5. An action to be called when the circuit changes to a Half-open state.

Executing a Delegate through the Policy

The overall operation is that the GetAsync method in the ResilientRequestProvider class invokes the HttpInvoker method, passing a Func that represents the GET request. The HttpInvoker method invokes Polly’s CircuitBreakerPolicy.ExecuteAsync method, passing the received Func as an argument.

The circuit breaker policy then attempts to execute the Func passed in via the ExecuteAsync method. The circuit initially starts closed and executes the Func, and measures its fault or success. If the number of consecutive faults exceeds a specified threshold (1), the circuit will break (move to an Open state).

When the circuit is in an Open state, any attempts to execute code through the policy will fail with a BrokenCircuitException being thrown. The circuit remains in an Open state for the duration specified by the CircuitBreakerAsync method (5s). After that timespan the circuit transitions to a Half-open state.

When the circuit is in a Half-open state, the delegate passed to the ExecuteAsync method will be attempted. If it succeeds the circuit transitions back to a Closed state. However, if the delegate throws a handled exception, the circuit transitions back to the Open state, remaining open for the configured timespan.

Summary

Polly is a .NET fault handling library, which includes fluent support for the circuit breaker pattern. In Polly, the circuit breaker pattern is implemented by the CircuitBreakerPolicy type, which handles specific exceptions thrown by, or results returned by, the delegates that are executed through the policy.

The CircuitBreakerPolicy type is highly configurable, allowing you to specify the exceptions to be handled, the returned results to be handled (not covered here), and how the policy should handle any faults.

The advantage of using Polly over implementing your own circuit breaker pattern is that Polly includes multiple fault handling patterns that can easily be combined for additional resilience when handling faults.

Fault Handling in Xamarin.Forms: Circuit Breaker

Calls to remote services can fail due to transient faults, such as the momentary loss of network connectivity to services, the temporary unavailability of a service, or timeouts that arise when the service is busy. These faults are often self-correcting, and if the remote access request is repeated after a suitable delay, it’s likely to succeed.

Earlier in the year I wrote about transient fault handling in Xamarin.Forms using the retry pattern. The idea being that all attempts to access a remote service can be wrapped in code that retries the operation if it fails. However, there can also be situations where faults are not transient. Instead, they are due to unanticipated events, and might take longer to fix. Such faults can range from a partial loss of connectivity to a complete service failure. In such circumstances it’s pointless for an application to continually retry an operation that’s unlikely to succeed. Instead, the app should accept that the operation has failed and act accordingly.

Faults that take a variable amount of time to recover from can be handled by the circuit breaker pattern, improving the stability and resiliency of an application.

Circuit Breaker Pattern

The circuit breaker pattern prevents an application from repeatedly trying to execute an operation that’s likely to fail. It monitors the number of recent failures that have occurred, and uses this information to decide whether to allow the operation to proceed, or whether to return an exception immediately. In addition, it also enables an application to detect whether the fault has been resolved, allowing the operation to be invoked again.

The pattern is so named because it sets states that mimic the functionality of an electrical circuit breaker:

• Closed. The remote access request is attempted. If the request fails a count of the number of recent failures is incremented. If this count exceeds a threshold within a time period, the circuit breaker is placed into Open state. A timeout timer then starts, to allow the problem that caused the failure to be fixed. When the timer expires the circuit breaker is placed into the Half-open state.
• Open. The remote access request fails immediately and an exception is returned to the application.
• Half-open. A limited number of remote access request are attempted. If the requests are successful, it’s assumed that the fault that was causing the failure has been fixed, and the circuit breaker is placed into the Closed state (while zeroing the failure counter). If any request fails it’s assumed that the fault is still present, and the circuit breaker is placed back into the Open state, where the timer restarts to allow the problem that caused the failure to be fixed.

Implementation

In this blog post I’ll explain how I implemented the circuit breaker pattern. Patterns & Practices describe an implementation of the pattern here, but don’t provide a fully working implementation. Therefore, my implementation is an attempt to flesh out the code they provided with a fully working implementation. The advantage of the approach presented here is that the circuit breaker pattern is implemented without requiring any library code, for those sensitive to bloating their application package size.

My implementation of the circuit breaker pattern adds to Xamarin’s TodoREST sample. This sample demonstrates a Todo list application where the data is stored and accessed from a RESTful web service, hosted by Xamarin. However, I’ve modified the original implementation so that the RestService class moves some of its responsibilities to the RequestProvider class, which handles all REST requests. This ensures that all REST requests are made by a single class, which has a single responsibility. The following code example shows the GetAsync method from the RequestProvider class, which makes GET requests to a specified URI:

public async Task<TResult> GetAsync<TResult>(string uri)
{
    var response = await client.GetAsync(uri);
    response.EnsureSuccessStatusCode();
    string serialized = await response.Content.ReadAsStringAsync();
    return JsonConvert.DeserializeObject<TResult>(serialized);
}

Note, however, that the sample application, which can be found on GitHub, doesn’t use the RequestProvider class. It’s included purely for comparison with the ResilientRequestProvider class, which the application uses, and which implements the circuit breaker pattern.

Initialization

The App class in the sample application initializes the classes that are responsible for communicating with the REST service:

TodoManager = new TodoItemManager(
    new RestService(
        new ResilientRequestProvider(
            new CircuitBreakerService(typeof(RestService).FullName, 1000))));

The RestService class provides data to the TodoItemManager, with the RestService class making REST calls using the ResilientRequestProvider class, which uses the CircuitBreakerService class to implement the circuit breaker pattern.

ResilientRequestProvider

The following code example shows the GetAsync method from the ResilientRequestProvider class, which makes GET requests to a specified URI:

async Task<HttpResponseMessage> HttpInvoker(Func<Task<HttpResponseMessage>> operation)
{
    return await circuitBreakerService.InvokeAsync(
        operation,
        // Perform a different operation when the breaker is open
        (circuitBreakerOpenException) => 
            Debug.WriteLine($"Circuit is open. Exception: {circuitBreakerOpenException.InnerException}"),
        // Different exception thrown
        (exception) => Debug.WriteLine($"Operation failed. Exception: {exception.Message}")
    );
}

public async Task<TResult> GetAsync<TResult>(string uri)
{
    string serialized = null;
    var httpResponse = await HttpInvoker(async () =>
    {
        var response = await client.GetAsync(uri);
        response.EnsureSuccessStatusCode();
        serialized = await response.Content.ReadAsStringAsync();
        return response;
    });
    return JsonConvert.DeserializeObject<TResult>(serialized);
}

Notice that the code from the GetAsync method in the RequestProvider class is still present, but is now specified as a lambda expression. This lambda expression is passed to the HttpInvoker method, which it turn passes it to the InvokeAsync method of the CircuitBreakerService class. Therefore, the code in the lambda expression is what will be executed by the circuit breaker, provided it’s in a state that allows that. In addition, the HttpInvoker method catches the CircuitBreakerOpenException if the operation fails because the circuit breaker is open.

CircuitBreakerService

The CircuitBreakerService class has a constructor with two arguments, as shown in the following code example:

public CircuitBreakerService(string resource, int openToHalfOpenWaitTime)
{
    _stateStore = CircuitBreakerStateStoreFactory.GetCircuitBreakerStateStore(resource);
    _resourceName = resource;
    _openToHalfOpenWaitTime = new TimeSpan(0, 0, 0, 0, openToHalfOpenWaitTime);
}

The constructor arguments specify a resource name that the circuit breaker is attempting to protect, and the time in milliseconds to wait when switching from the Open to Half-open state (the length of time the circuit breaker waits for the fault to be fixed).

The InvokeAsync method in the CircuitBreakerService class is shown in the following code example:

public async Task<HttpResponseMessage> InvokeAsync(Func<Task<HttpResponseMessage>> operation)
{
    HttpResponseMessage response = null;

    if (IsOpen)
    {
        // Circuit breaker is open
        return await WhenCircuitIsOpenAsync(operation);
    }
    else
    {
        // Circuit breaker is closed - execute the operation
        try
        {
            response = await operation();
        }
        catch (Exception ex)
        {
            // Retrip the breaker immediately and throw the exception so that
            // the caller can tell the type of exception that was thrown
            TrackException(ex);
            throw;
        }
    }
    return response;
}

public async Task<HttpResponseMessage> InvokeAsync(
    Func<Task<HttpResponseMessage>> operation, 
    Action<CircuitBreakerOpenException> circuitBreakerOpenAction, 
    Action<Exception> anyOtherExceptionAction)
{
    HttpResponseMessage response = null;

    try
    {
        response = await InvokeAsync(operation);
    }
    catch (CircuitBreakerOpenException ex)
    {
        // Perform a different operation when the circuit breaker is open
        circuitBreakerOpenAction(ex);
    }
    catch (Exception ex)
    {
        anyOtherExceptionAction(ex);
    }
    return response;
}

The first InvokeAsync method wraps an operation, specified as a Func. If the circuit breaker is closed, it invokes the Func. If the operation fails, an exception handler calls the TrackException method, which sets the circuit breaker state to Open. The second InvokeAsync method wraps an operation, specified as a Func, and an Action to be performed if a CircuitBreakerOpenException is thrown, and an Action to be performed when any other exception is thrown.

If the circuit breaker is in an Open state, the InvokeAsync method calls the WhenCircuitIsOpenAsync, which is shown in the following code example:

async Task<HttpResponseMessage> WhenCircuitIsOpenAsync(Func<Task<HttpResponseMessage>> operation)
{
    HttpResponseMessage response = null;

    if (_stateStore.LastStateChangedDate + _openToHalfOpenWaitTime < DateTime.UtcNow)
    {
        bool lockTaken = false;
        try
        {
            Monitor.TryEnter(_halfOpenSyncObject, ref lockTaken);
            if (lockTaken)
            {
                _stateStore.HalfOpen();
                response = await operation();
                _stateStore.Reset();
                return response;
            }
        }
        catch (Exception ex)
        {
            _stateStore.Trip(ex);
            throw;
        }
        finally
        {
            if (lockTaken)
            {
                Monitor.Exit(_halfOpenSyncObject);
            }
        }
    }
}

This method first checks if the circuit breaker open timeout has expired. If this is the case, the circuit breaker is set to a Half-open state, and then the operation specified by the Func is performed. If the operation is successful, the circuit breaker is reset to the Closed state. If the operation fails, it is tripped back to the Open state and the time the exception occurred is updated so that the circuit breaker will wait for a further period before trying to perform the operation again.

If the circuit breaker has only been in an Open state for a short time (the open timeout hasn’t expired), the method throws a CircuitBreakerOpenException and returns the error that caused the circuit breaker to transition to the Open state.

Note that the WhenCircuitIsOpenAsync method uses a lock to prevent the circuit breaker from trying to perform concurrent calls to the operation while it’s Half-open. A concurrent attempt to invoke the operation will be handled as if the circuit breaker was Open, and it’ll fail with a CircuitBreakerOpenException.

Running the Sample Application

The sample application, which can be found on GitHub, connects to a read-only REST service hosted by Xamarin, and it’s most likely that when running the sample the GET operation will succeed on first attempt. To observe the circuit breaker pattern in operation, change the RestUrl property in the Constants class to an address that doesn’t exist – this can be accomplished by adding a random character to the existing string. Then run the application and observe the output in the output window in Visual Studio. You should see something like:

Circuit is closed. Executing operation.
Tripping the circuit breaker.
Operation failed. Exception: 404 (Not Found)
ERROR: Value cannot be null.
Parameter name: value

This shows attempted execution of the GET operation through the circuit breaker. Initially the circuit is Closed and so an attempt is made to execute the GET operation. The operation fails and so the circuit breaker is tripped, and the error message is presented to the user. Remember that the length of the circuit breaker open timeout can be specified through the CircuitBreakerService constructor. This allows the circuit breaker pattern to be customized to fit individual application requirements.

Summary

The circuit breaker pattern prevents an application from repeatedly trying to execute an operation that’s likely to fail. The pattern also enables an application to detect whether the fault has been resolved, allowing the operation to be invoked again. The pattern monitors the number of recent failures that have occurred, and uses this information to decide whether to allow the operation to proceed, or whether to return an exception immediately. It does this by setting states that mimic the functionality of an electrical circuit breaker.

This blog post has described an implementation of the circuit breaker pattern, based on descriptions provided by Patterns & Practices. The length of the circuit breaker open timeout can be specified, allowing the pattern to be customized to fit individual application requirements. The advantage of the approach presented here is that the circuit breaker pattern is implemented without requiring any library code, for those sensitive to bloating their application package size.

In my next blog post I’ll show how to re-implement the circuit breaker pattern using Polly.

Using PKCE with IdentityServer from a Xamarin Client

The OpenID Connect and OAuth 2.0 specifications define a number of authentication flows between clients and authentication providers. These include:

• Implicit.This authentication flow is optimized for browser-based apps. All tokens are transmitted via the browser.
• Authorization code. This authentication flow provides the ability to retrieve tokens on a back channel, as opposed to the browser front channel, while also supporting client authentication.
• Hybrid. This authentication flow is a combination of the implicit and authorization code flows. The identity token is transmitted via the browser channel and contains the signed protocol response along with other artifacts such as the authorization code. After successful validation of the response, the back channel is used to retrieve the access and refresh tokens.

The eShopOnContainers mobile app communicates with an identity microservice, which uses IdentityServer 4 to perform authentication, and access control for APIs. The app uses the hybrid authentication flow to retrieve access tokens, as this flow mitigates a number of attacks that apply to the browser channel, and this approach is explained in the guidance documentation.

However, OAuth 2.0 clients that utilize authorization codes are susceptible to an authorization code interception attack. In this attack, the authorization code returned from an authorization endpoint is intercepted within a communication path not protected by Transport Layer Security (TLS), such as inter-application communication within the client’s operating system. Once the attacker has gained access to the authorization code, it can be used to obtain the access token. While a number of pre-conditions must hold for the authorization code interception attack to work, it has been observed in the wild.

To mitigate this attack, the Proof Key for Code Exchange (PKCE) extension to OAuth 2.0 adds additional parameters to the OAuth 2.0 authorization and access token requests:

1. The client creates a cryptographically random key called a code verifier, and derives a transformed value, called a code challenge, which is sent in the OAuth 2.0 authorization request along with the transformation method.
2. The authorization endpoint responds as usual but records the code challenge and transformation method.
3. The clients sends the authorization code in the access token request, and also includes the code verifier.
4. The authorization server transforms the code verifier and compares it to the code challenge. Access is denied if they are not equal.

This works as a mitigation for native apps because if an attacker intercepts the authorization code in step (2), it can’t redeem it for an access token as the attacker is not in possession of the code verifier. In addition, the code verifier can’t be intercepted since it’s sent over TLS.

For detailed information about PKCE, see Proof Key for Code Exchange by OAuth Public Clients.

Implementing PKCE

Following the guidance in the OAuth 2.0 for Native Apps specification, that PKCE should be used in authorization code based authentication flows, I’ve recently updated the eShopOnContainers mobile app to use PKCE when communicating with IdentityServer.

Server Side

IdentityServer must be configured to require the use of PKCE. This is achieved by modifying the configuration of the IdentityServer Client object for the Xamarin client:

public static IEnumerable<Client> GetClients(Dictionary<string,string> clientsUrl)
{
    return new List<Client>
    {
        ...
        new Client
        {
            ClientId = "xamarin",
            ClientName = "eShop Xamarin OpenId Client",
            AllowedGrantTypes = GrantTypes.Hybrid,
            ClientSecrets =
            {
                new Secret("secret".Sha256())
            },
            RedirectUris = { clientsUrl["Xamarin"] },
            RequireConsent = false,
            RequirePkce = true,
            PostLogoutRedirectUris = { $"{clientsUrl["Xamarin"]}/Account/Redirecting" },
            AllowedCorsOrigins = { "http://eshopxamarin" },
            AllowedScopes = new List<string>
            {
                IdentityServerConstants.StandardScopes.OpenId,
                IdentityServerConstants.StandardScopes.Profile,
                IdentityServerConstants.StandardScopes.OfflineAccess,
                "orders",
                "basket"
            },
            AllowOfflineAccess = true,
            AllowAccessTokensViaBrowser = true
        },
        ...
    };
}

This configuration adds the RequirePkce property to the Client object. The RequirePkce property specifies whether clients using an authorization code must send a proof key.

Client Side

The CreateAuthorizationRequest method in the IdentityService class creates the URI for IdentityServer’s authorization endpoint, and the URI must be modified to include additional query parameters. The following code example shows the modified method:

public string CreateAuthorizationRequest()
{
    // Create URI to authorization endpoint
    var authorizeRequest = new AuthorizeRequest(GlobalSetting.Instance.IdentityEndpoint);

    // Dictionary with values for the authorize request
    var dic = new Dictionary<string, string>();
    dic.Add("client_id", GlobalSetting.Instance.ClientId);
    dic.Add("client_secret", GlobalSetting.Instance.ClientSecret); 
    dic.Add("response_type", "code id_token");
    dic.Add("scope", "openid profile basket orders locations marketing offline_access");
    dic.Add("redirect_uri", GlobalSetting.Instance.IdentityCallback);
    dic.Add("nonce", Guid.NewGuid().ToString("N"));
    dic.Add("code_challenge", CreateCodeChallenge());
    dic.Add("code_challenge_method", "S256");

    // Add CSRF token to protect against cross-site request forgery attacks.
    var currentCSRFToken = Guid.NewGuid().ToString("N");
    dic.Add("state", currentCSRFToken);

    var authorizeUri = authorizeRequest.Create(dic); 
    return authorizeUri;
}

The client first creates a code verifier for the authorization request, with the CreateCodeChallenge method:

private string CreateCodeChallenge()
{
    _codeVerifier = RandomNumberGenerator.CreateUniqueId();
    var sha256 = HashAlgorithmProvider.OpenAlgorithm(HashAlgorithm.Sha256);
    var challengeBuffer = sha256.HashData(
        CryptographicBuffer.CreateFromByteArray(Encoding.UTF8.GetBytes(_codeVerifier)));
    byte[] challengeBytes;
    CryptographicBuffer.CopyToByteArray(challengeBuffer, out challengeBytes);
    return Base64Url.Encode(challengeBytes);
}

The CreateUniqueId method in the RandomNumberGenerator class creates a high-entropy cryptographic random string using the PCLCrypto library. Note that the PKCE specification requires that the code verifier is base64 URL-encoded to produce a URL safe string. However, the code verifier here is already URL safe, and so this additional operation isn’t required.

The CreateCodeChallenge method then creates a code challenge derived from the code verifier. This can be achieved by using one of the following transformations:

• code challenge = code verifier (known as the plain transformation)

OR

• code challenge = base64urlencode(Sha256(code_verifier)) (known as the S256 transformation)

If the client is capable of using the S256 transformation, it must do so, as this transformation is mandatory to implement on compliant servers. The CreateAuthorizationRequest method uses the S256 transformation, which SHA256 encodes the code verifier, and then base64 url-encodes the SHA256 output.

The client then sends the code challenge as part of the OAuth 2.0 authorization request, using the following additional parameters:

• code_challenge – the derived code challenge
• code_challenge_method – S256 (or plain)

When IdentityServer issues the authorization code in the authorization response, it associates the code challenge and code challenge method values with the authorization code so that it can be verified later. Note that if IdentityServer is configured to use PKCE, and the client does not send the code challenge, the authorization endpoint responds with an error response set to invalid_request.

Upon receipt of the authorization code, the client sends the access token request to the token endpoint. In addition to the existing parameters, it also sends the following parameter:

• code_verifier – the code verifier

In the eShopOnContainers app, this is achieved with the GetTokenAsync method in the IdentityService class:

public async Task<UserToken> GetTokenAsync(string code)
{
    var data = string.Format("grant_type=authorization_code&code={0}&redirect_uri={1}&code_verifier={2}", 
        code, WebUtility.UrlEncode(GlobalSetting.Instance.IdentityCallback), _codeVerifier);
    var token = await _requestProvider.PostAsync<UserToken>(GlobalSetting.Instance.TokenEndpoint, 
        data, GlobalSetting.Instance.ClientId, GlobalSetting.Instance.ClientSecret);
    return token;
}

Upon receipt of the request at the token endpoint, IdentityServer verifies it by calculating the code challenge from the received code verifier, and comparing it with the previously associated code challenge, after first transforming it according to the code challenge method specified by the client. If the values are not equal, an error response indicating invalid_grant is returned. If the values are equal, the token endpoint continues processing as normal and responds with an access token, identity token, and refresh token.

Summary

OAuth 2.0 clients that utilize authorization codes are susceptible to an authorization code interception attack. To mitigate this attack, the PKCE extension to OAuth 2.0 adds additional parameters to the OAuth 2.0 authorization and access token requests.

Following the guidance in the OAuth 2.0 for Native Apps specification, that PKCE should be used in authorization code based authentication flows, I’ve recently updated the eShopOnContainers mobile app to use PKCE when communicating with IdentityServer.

For detailed information about PKCE, see Proof Key for Code Exchange by OAuth Public Clients.