Tuesday, 15 December 2015

Accessing Image Pixel Data in a Xamarin.Forms App – Android

Previously, I’ve described a basic architecture for accessing and manipulating pixel data from a Xamarin.Forms project. A Xamarin.Forms PCL project defines the IBitmap interface, which specifies the operations that must be implemented in each platform-specific project to access and manipulate pixel data. I then explained the operation of the Bitmap class in the Xamarin.Forms.iOS project, which implements the IBitmap interface.

In this blog post I’ll explain the operation of the Bitmap class in the Xamarin.Forms Android project, which implements the IBitmap interface. The DependencyService can be used to invoke a method from the Xamarin.Forms PCL project, which in turn invokes IBitmap operations.

Disclaimer: The code featured here is alpha code, so all the usual caveats apply.

Implementation

The following code shows the Bitmap class implemented on Android:

1 public class Bitmap : IBitmap<Android.Graphics.Bitmap> 2 { 3 const byte bytesPerPixel = 4; 4 readonly int height; 5 readonly int width; 6 byte[] pixelData; 7 Android.Graphics.Bitmap bitmap; 8 string photoFile; 9 10 public Bitmap (string photo) 11 { 12 photoFile = photo; 13 var options = new BitmapFactory.Options { 14 InJustDecodeBounds = true 15 }; 16 17 // Bitmap will be null because InJustDecodeBounds = true 18 bitmap = BitmapFactory.DecodeFile (photoFile, options); 19 width = options.OutWidth; 20 height = options.OutHeight; 21 } 22 23 public void ToPixelArray () 24 { 25 bitmap = BitmapFactory.DecodeFile (photoFile); 26 27 int size = width * height * bytesPerPixel; 28 pixelData = new byte[size]; 29 var byteBuffer = Java.Nio.ByteBuffer.AllocateDirect (size); 30 bitmap.CopyPixelsToBuffer (byteBuffer); 31 Marshal.Copy (byteBuffer.GetDirectBufferAddress (), pixelData, 0, size); 32 byteBuffer.Dispose (); 33 } 34 35 public void TransformImage (Func<byte, byte, byte, double> pixelOperation) 36 { 37 byte r, g, b; 38 39 try { 40 // Pixel data order is RGBA 41 for (int i = 0; i < pixelData.Length; i += bytesPerPixel) { 42 r = pixelData [i]; 43 g = pixelData [i + 1]; 44 b = pixelData [i + 2]; 45 46 // Leave alpha value intact 47 pixelData [i] = pixelData [i + 1] = pixelData [i + 2] = (byte)pixelOperation (r, g, b); 48 } 49 } catch (Exception ex) { 50 Console.WriteLine (ex.Message); 51 } 52 } 53 54 public Android.Graphics.Bitmap ToImage () 55 { 56 var byteBuffer = Java.Nio.ByteBuffer.AllocateDirect (width * height * bytesPerPixel); 57 Marshal.Copy (pixelData, 0, byteBuffer.GetDirectBufferAddress (), width * height * bytesPerPixel); 58 bitmap.CopyPixelsFromBuffer (byteBuffer); 59 byteBuffer.Dispose (); 60 return bitmap; 61 } 62 63 public void Dispose () 64 { 65 if (bitmap != null) { 66 bitmap.Recycle (); 67 bitmap.Dispose (); 68 bitmap = null; 69 } 70 pixelData = null; 71 } 72 }

The constructor reads and decodes the photo. Setting the InJustDecodeBounds property to true while decoding avoids allocating memory for the bitmap and hence returns null for the bitmap object, but does set the OutWidth, and OutHeight properties. This technique allows you to read the dimensions and type of the image data prior to construction (and memory allocation) of the bitmap.

The ToPixelArray method is used to read the photo to a Android.Graphics.Bitmap and convert it to a byte array of raw pixel data. The image’s pixel data is placed into a continuous sequence of bytes. It’s formatted as 32 bits per pixel, with each pixel being composed of a Red, Green, Blue, and Alpha channel. It’s then copied to the pixelData array. Therefore, the pixelData array stores each pixel over four array elements (R, G, B, A). It’s likely that the pixelData array is not required, and that the pixel data can be accessed directly through the ByteBuffer instance. I’ll be exploring this performance optimization at a later date.

The TransformImage method is used to perform an imaging operation on the pixel data, such as convert to greyscale. The method specifies a Func parameter that’s used to perform a per-pixel operation. The Func is passed from the Xamarin.Forms PCL project into the method, and operates on each pixel component. At the moment the TransformImage method is hard-coded to place the result of the Func into each pixel component. This is purely for my own test purposes, for implementing a ConvertToGreyscale Func. It can easily be modified to not duplicate the result of the Func across all four colour components.

The ToImage method is then used to place the transformed pixel data back into the Android.Graphics.Bitmap. Note that the ToImage method creates a 32 bit image. An overload could be provided to create a 8 bit image.

A Dispose method is also provided to clean up the Android.Graphics.Bitmap instance and byte array of pixels.

The Bitmap class can then be invoked as follows:

1 public Task<Stream> TransformPhotoAsync (Func<byte, byte, byte, double> pixelOperation) 2 { 3 return Task.Run (() => { 4 var bitmap = new Bitmap (photoPath); 5 bitmap.ToPixelArray (); 6 bitmap.TransformImage (pixelOperation); 7 var androidBitmap = bitmap.ToImage (); 8 9 var memoryStream = new MemoryStream (); 10 androidBitmap.Compress (Android.Graphics.Bitmap.CompressFormat.Jpeg, 80, memoryStream); 11 memoryStream.Seek (0L, SeekOrigin.Begin); 12 bitmap.Dispose (); 13 14 return (Stream)memoryStream; 15 }); 16 }

A new Bitmap instance is created, with the path to the photo being passed into the constructor. The photo is read and converted to an array of pixels, transformed by the pixelOperation Func, and then converted into a new Android.Graphics.Bitmap before being returned for display by a Xamarin.Forms Image control. All of this work is wrapped in a Task.Run in order for it to be executed on a background thread.

Summary

This blog post has described how to access and manipulate pixel data in a Xamarin.Forms Android project, by implementing the Bitmap class. The Bitmap class implements the IBitmap interface that’s provided by the Xamarin.Forms PCL project. My next blog post will explain the implementation of the IBitmap interface in WinRT.

3 comments:

  1. I'm trying to work through your articles on implementing iBitmap in the PCL, Xamarin.Android, Xamarin.iO. Some details are eluding me.

    1) Why does droid ToPixelArray() re-read the file from the filesystem to set the bitmap field? Didn't that already happen in the constructor? If there is concern about it being disposed of someplace else, couldn't it first be checked if null?
    if (bitmap==null) bitmap = BitmapFactory.DecodeFile(photoFile);

    2) How does a class in the PCL actually make calls to DependecyService.Get if IBitmap has to specify a type which can only exist in the device code (Android.Bitmap, iOS UIView etc)? By that I mean you can't do this in the PCL:
    DependencyService.Get>().TransformImage(ImagingOperations.ConvertToGreyscale);
    because the PCL doesn't have a Bitmap type to supply as T

    Or am I just wrong in my thinking that could do this in a class in the PCL:
    void ProcessMyPhoto()
    {
    DependencyService.Get>().TransformImage(ImagingOperations.ScaleByHalf);
    DependencyService.Get>().TransformImage(ImagingOperations.Rotate180);
    DependencyService.Get>().TransformImage(ImagingOperations.ConvertToGreyscale);
    DependencyService.Get>().TransformImage(ImagingOperations.Save);
    }
    I thought that is the point though, to keep the logic at the shared level, and offload the work to the individual device specific implementations that understand their concept of what a bitmap is.

    ReplyDelete
  2. 1. The constructor doesn't read the entire file. It just opens the file and reads the width and height. That's what setting InJustDecodeBounds = true enables. Reference: https://developer.xamarin.com/recipes/android/resources/general/load_large_bitmaps_efficiently/#Read_Bitmap_Dimensions_and_Type

    2. You can use the DependencyService to invoke all this from a PCL. The code that constructs a Bitmap object, and uses it, will exist in your platform-specific projects. So my PCL does:

    var stream = await photoOperation.TransformPhotoAsync (ImagingOperations.ConvertToGreyscale);

    Where photoOperation has previously been retrieved by the DependencyService.

    Then my Android PhotoOperation class is:

    public class PhotoOperation : IPhotoOperation
    {
    bool isDirty;

    public static string PhotoPath { get; set; }

    public Task TransformPhotoAsync (Func pixelOperation)
    {
    return Task.Run (async () => {
    var bitmap = new Bitmap (PhotoPath);
    bitmap.ToPixelArray ();
    bitmap.TransformImage (pixelOperation);
    var androidBitmap = bitmap.ToImage ();

    var memoryStream = new MemoryStream ();
    await androidBitmap.CompressAsync (Android.Graphics.Bitmap.CompressFormat.Png, 100, memoryStream);
    memoryStream.Seek (0L, SeekOrigin.Begin);
    bitmap.Dispose ();
    isDirty = true;

    return (Stream)memoryStream;
    });
    }
    }

    Sorry I haven't shared a downloadable solution for all this, but it's very much work in progress and will see the light of day eventually. If you need any more help send me your email address (contact form or twitter) and we can take it up over email.

    ReplyDelete
  3. Just wanted to say "Thanks for the fast response!" That implementation example will, I think, clear it up in my mind after I take a bit to work with it and wrap my mind around it. Well, know I know how I'm spending my evening .

    ReplyDelete