Calls to remote services can fail due to transient faults, such as the momentary loss of network connectivity to services, the temporary unavailability of a service, or timeouts that arise when the service is busy. These faults are often self-correcting, and if the remote access request is repeated after a suitable delay, it’s likely to succeed.
Earlier in the year I wrote about transient fault handling in Xamarin.Forms using the retry pattern. The idea being that all attempts to access a remote service can be wrapped in code that retries the operation if it fails. However, there can also be situations where faults are not transient. Instead, they are due to unanticipated events, and might take longer to fix. Such faults can range from a partial loss of connectivity to a complete service failure. In such circumstances it’s pointless for an application to continually retry an operation that’s unlikely to succeed. Instead, the app should accept that the operation has failed and act accordingly.
Faults that take a variable amount of time to recover from can be handled by the circuit breaker pattern, improving the stability and resiliency of an application.
Circuit Breaker Pattern
The circuit breaker pattern prevents an application from repeatedly trying to execute an operation that’s likely to fail. It monitors the number of recent failures that have occurred, and uses this information to decide whether to allow the operation to proceed, or whether to return an exception immediately. In addition, it also enables an application to detect whether the fault has been resolved, allowing the operation to be invoked again.
The pattern is so named because it sets states that mimic the functionality of an electrical circuit breaker:
- Closed. The remote access request is attempted. If the request fails a count of the number of recent failures is incremented. If this count exceeds a threshold within a time period, the circuit breaker is placed into Open state. A timeout timer then starts, to allow the problem that caused the failure to be fixed. When the timer expires the circuit breaker is placed into the Half-open state.
- Open. The remote access request fails immediately and an exception is returned to the application.
- Half-open. A limited number of remote access request are attempted. If the requests are successful, it’s assumed that the fault that was causing the failure has been fixed, and the circuit breaker is placed into the Closed state (while zeroing the failure counter). If any request fails it’s assumed that the fault is still present, and the circuit breaker is placed back into the Open state, where the timer restarts to allow the problem that caused the failure to be fixed.
Implementation
In this blog post I’ll explain how I implemented the circuit breaker pattern. Patterns & Practices describe an implementation of the pattern here, but don’t provide a fully working implementation. Therefore, my implementation is an attempt to flesh out the code they provided with a fully working implementation. The advantage of the approach presented here is that the circuit breaker pattern is implemented without requiring any library code, for those sensitive to bloating their application package size.
My implementation of the circuit breaker pattern adds to Xamarin’s TodoREST sample. This sample demonstrates a Todo list application where the data is stored and accessed from a RESTful web service, hosted by Xamarin. However, I’ve modified the original implementation so that the RestService class moves some of its responsibilities to the RequestProvider class, which handles all REST requests. This ensures that all REST requests are made by a single class, which has a single responsibility. The following code example shows the GetAsync method from the RequestProvider class, which makes GET requests to a specified URI:
public async Task<TResult> GetAsync<TResult>(string uri)
{
var response = await client.GetAsync(uri);
response.EnsureSuccessStatusCode();
string serialized = await response.Content.ReadAsStringAsync();
return JsonConvert.DeserializeObject<TResult>(serialized);
}
Note, however, that the sample application, which can be found on GitHub, doesn’t use the RequestProvider class. It’s included purely for comparison with the ResilientRequestProvider class, which the application uses, and which implements the circuit breaker pattern.
Initialization
The App class in the sample application initializes the classes that are responsible for communicating with the REST service:
TodoManager = new TodoItemManager(
new RestService(
new ResilientRequestProvider(
new CircuitBreakerService(typeof(RestService).FullName, 1000))));
The RestService class provides data to the TodoItemManager, with the RestService class making REST calls using the ResilientRequestProvider class, which uses the CircuitBreakerService class to implement the circuit breaker pattern.
ResilientRequestProvider
The following code example shows the GetAsync method from the ResilientRequestProvider class, which makes GET requests to a specified URI:
async Task<HttpResponseMessage> HttpInvoker(Func<Task<HttpResponseMessage>> operation)
{
return await circuitBreakerService.InvokeAsync(
operation,
// Perform a different operation when the breaker is open
(circuitBreakerOpenException) =>
Debug.WriteLine($"Circuit is open. Exception: {circuitBreakerOpenException.InnerException}"),
// Different exception thrown
(exception) => Debug.WriteLine($"Operation failed. Exception: {exception.Message}")
);
}
public async Task<TResult> GetAsync<TResult>(string uri)
{
string serialized = null;
var httpResponse = await HttpInvoker(async () =>
{
var response = await client.GetAsync(uri);
response.EnsureSuccessStatusCode();
serialized = await response.Content.ReadAsStringAsync();
return response;
});
return JsonConvert.DeserializeObject<TResult>(serialized);
}
Notice that the code from the GetAsync method in the RequestProvider class is still present, but is now specified as a lambda expression. This lambda expression is passed to the HttpInvoker method, which it turn passes it to the InvokeAsync method of the CircuitBreakerService class. Therefore, the code in the lambda expression is what will be executed by the circuit breaker, provided it’s in a state that allows that. In addition, the HttpInvoker method catches the CircuitBreakerOpenException if the operation fails because the circuit breaker is open.
CircuitBreakerService
The CircuitBreakerService class has a constructor with two arguments, as shown in the following code example:
public CircuitBreakerService(string resource, int openToHalfOpenWaitTime)
{
_stateStore = CircuitBreakerStateStoreFactory.GetCircuitBreakerStateStore(resource);
_resourceName = resource;
_openToHalfOpenWaitTime = new TimeSpan(0, 0, 0, 0, openToHalfOpenWaitTime);
}
The constructor arguments specify a resource name that the circuit breaker is attempting to protect, and the time in milliseconds to wait when switching from the Open to Half-open state (the length of time the circuit breaker waits for the fault to be fixed).
The InvokeAsync method in the CircuitBreakerService class is shown in the following code example:
public async Task<HttpResponseMessage> InvokeAsync(Func<Task<HttpResponseMessage>> operation)
{
HttpResponseMessage response = null;
if (IsOpen)
{
// Circuit breaker is open
return await WhenCircuitIsOpenAsync(operation);
}
else
{
// Circuit breaker is closed - execute the operation
try
{
response = await operation();
}
catch (Exception ex)
{
// Retrip the breaker immediately and throw the exception so that
// the caller can tell the type of exception that was thrown
TrackException(ex);
throw;
}
}
return response;
}
public async Task<HttpResponseMessage> InvokeAsync(
Func<Task<HttpResponseMessage>> operation,
Action<CircuitBreakerOpenException> circuitBreakerOpenAction,
Action<Exception> anyOtherExceptionAction)
{
HttpResponseMessage response = null;
try
{
response = await InvokeAsync(operation);
}
catch (CircuitBreakerOpenException ex)
{
// Perform a different operation when the circuit breaker is open
circuitBreakerOpenAction(ex);
}
catch (Exception ex)
{
anyOtherExceptionAction(ex);
}
return response;
}
The first InvokeAsync method wraps an operation, specified as a Func. If the circuit breaker is closed, it invokes the Func. If the operation fails, an exception handler calls the TrackException method, which sets the circuit breaker state to Open. The second InvokeAsync method wraps an operation, specified as a Func, and an Action to be performed if a CircuitBreakerOpenException is thrown, and an Action to be performed when any other exception is thrown.
If the circuit breaker is in an Open state, the InvokeAsync method calls the WhenCircuitIsOpenAsync, which is shown in the following code example:
async Task<HttpResponseMessage> WhenCircuitIsOpenAsync(Func<Task<HttpResponseMessage>> operation)
{
HttpResponseMessage response = null;
if (_stateStore.LastStateChangedDate + _openToHalfOpenWaitTime < DateTime.UtcNow)
{
bool lockTaken = false;
try
{
Monitor.TryEnter(_halfOpenSyncObject, ref lockTaken);
if (lockTaken)
{
_stateStore.HalfOpen();
response = await operation();
_stateStore.Reset();
return response;
}
}
catch (Exception ex)
{
_stateStore.Trip(ex);
throw;
}
finally
{
if (lockTaken)
{
Monitor.Exit(_halfOpenSyncObject);
}
}
}
}
This method first checks if the circuit breaker open timeout has expired. If this is the case, the circuit breaker is set to a Half-open state, and then the operation specified by the Func is performed. If the operation is successful, the circuit breaker is reset to the Closed state. If the operation fails, it is tripped back to the Open state and the time the exception occurred is updated so that the circuit breaker will wait for a further period before trying to perform the operation again.
If the circuit breaker has only been in an Open state for a short time (the open timeout hasn’t expired), the method throws a CircuitBreakerOpenException and returns the error that caused the circuit breaker to transition to the Open state.
Note that the WhenCircuitIsOpenAsync method uses a lock to prevent the circuit breaker from trying to perform concurrent calls to the operation while it’s Half-open. A concurrent attempt to invoke the operation will be handled as if the circuit breaker was Open, and it’ll fail with a CircuitBreakerOpenException.
Running the Sample Application
The sample application, which can be found on GitHub, connects to a read-only REST service hosted by Xamarin, and it’s most likely that when running the sample the GET operation will succeed on first attempt. To observe the circuit breaker pattern in operation, change the RestUrl property in the Constants class to an address that doesn’t exist – this can be accomplished by adding a random character to the existing string. Then run the application and observe the output in the output window in Visual Studio. You should see something like:
Circuit is closed. Executing operation.
Tripping the circuit breaker.
Operation failed. Exception: 404 (Not Found)
ERROR: Value cannot be null.
Parameter name: value
This shows attempted execution of the GET operation through the circuit breaker. Initially the circuit is Closed and so an attempt is made to execute the GET operation. The operation fails and so the circuit breaker is tripped, and the error message is presented to the user. Remember that the length of the circuit breaker open timeout can be specified through the CircuitBreakerService constructor. This allows the circuit breaker pattern to be customized to fit individual application requirements.
Summary
The circuit breaker pattern prevents an application from repeatedly trying to execute an operation that’s likely to fail. The pattern also enables an application to detect whether the fault has been resolved, allowing the operation to be invoked again. The pattern monitors the number of recent failures that have occurred, and uses this information to decide whether to allow the operation to proceed, or whether to return an exception immediately. It does this by setting states that mimic the functionality of an electrical circuit breaker.
This blog post has described an implementation of the circuit breaker pattern, based on descriptions provided by Patterns & Practices. The length of the circuit breaker open timeout can be specified, allowing the pattern to be customized to fit individual application requirements. The advantage of the approach presented here is that the circuit breaker pattern is implemented without requiring any library code, for those sensitive to bloating their application package size.
In my next blog post I’ll show how to re-implement the circuit breaker pattern using Polly.