Adventures in ARKit - image detection

In my previous blog post I discussed how to display a 3D model in a scene. In this blog I’ll discuss how to perform image detection in a scene. Specifically, the app will identify the following image in a scene, and highlight it:

The sample this code comes from can be found on GitHub.

Image detection

The simplest approach to declaring the image to be detected is to add it to your app’s asset catalog as an AR Reference Image inside an AR Resource Group.

Writing code to detect the image is a two-step process:

1. Create an ARSCNViewDelegate class that defines the code to be executed when the image is detected.
2. Consume the ARSCNViewDelegate instance in your ViewController class, to detect the image.

The following code example shows the SceneViewDelegate class, which derives from ARSCNViewDelegate:

using System;
using ARKit;
using ARKitFun.Nodes;
using SceneKit;
using UIKit;

namespace ARKitFun
{
    public class SceneViewDelegate : ARSCNViewDelegate
    {
        public override void DidAddNode(ISCNSceneRenderer renderer, SCNNode node, ARAnchor anchor)
        {
            if (anchor is ARImageAnchor imageAnchor)
            {
                ARReferenceImage image = imageAnchor.ReferenceImage;
                nfloat width = image.PhysicalSize.Width;
                nfloat height = image.PhysicalSize.Height;

                PlaneNode planeNode = new PlaneNode(width, height, new SCNVector3(0, 0, 0), UIColor.Red);
                float angle = (float)(-Math.PI / 2);
                planeNode.EulerAngles = new SCNVector3(angle, 0, 0);
                node.AddChildNode(planeNode);
            }
        }
    }
}

The SceneViewDelegate class overrides the DidAddNode method, which is executed when the image is detected in the scene. This method first checks that the detected image is an ARImageAnchor, which represents an anchor for a known image that ARKit detects in the scene. Then the dimensions of the detected image are determined, and a red PlaneNode (of the same dimensions) is created and overlaid on the detected image. In addition, the overlaid PlaneNode will always orient itself correctly over the detected image.

The PlaneNode class is simply an SCNNode, which uses an SCNPlane geometry that represents a square or rectangle:

using System;
using SceneKit;
using UIKit;

namespace ARKitFun.Nodes
{
    public class PlaneNode : SCNNode
    {
        public PlaneNode(nfloat width, nfloat length, SCNVector3 position, UIColor color)
        {
            SCNNode node = new SCNNode
            {
                Geometry = CreateGeometry(width, length, color),
                Position = position,
                Opacity = 0.5f
            };

            AddChildNode(node);
        }

        SCNGeometry CreateGeometry(nfloat width, nfloat length, UIColor color)
        {
            SCNMaterial material = new SCNMaterial();
            material.Diffuse.Contents = color;
            material.DoubleSided = false;

            SCNPlane geometry = SCNPlane.Create(width, length);
            geometry.Materials = new[] { material };

            return geometry;
        }
    }
}

The PlaneNode constructor takes arguments that represent the width and height of the node, it’s position, and a color. The constructor creates a SCNNode, assigns a geometry to its Geometry property, sets its position and opacity, and adds the child node to the SCNNode.

The SceneViewDelegate class can then be consumed in your ViewController class, to detect the image:

using System;
using ARKit;
using Foundation;
using UIKit;

namespace ARKitFun
{
    public partial class ViewController : UIViewController
    {
        readonly ARSCNView sceneView;

        public ViewController(IntPtr handle) : base(handle)
        {
            sceneView = new ARSCNView
            {
                ShowsStatistics = true,
                Delegate = new SceneViewDelegate()
            };
            View.AddSubview(sceneView);
        }

        public override void ViewDidAppear(bool animated)
        {
            base.ViewDidAppear(animated);

            NSSet<ARReferenceImage> images = ARReferenceImage.GetReferenceImagesInGroup("AR Resources", null);

            sceneView.Session.Run(new ARWorldTrackingConfiguration
            {
                AutoFocusEnabled = true,
                LightEstimationEnabled = true,
                DetectionImages = images
            }, ARSessionRunOptions.ResetTracking | ARSessionRunOptions.RemoveExistingAnchors);
        }
        ...
    }
}

The ViewController constructor creates an instance of the SceneViewDelegate class and sets the instance as the Delegate property of the ARSCNView. In addition, the ViewDidAppear method is modified to retrieve the image to be detected from the asset catalog, and set it as the DetectionImages property of the ARWorldTrackingConfiguration object.

The overall effect is that when the image is detected in the scene, a red rectangle is overlaid on it:

Then, the red rectangle reorients itself in realtime if the orientation of the detected image in the scene changes:

Once an object has been identified in a scene, it can be manipulated, and this will be what I explore in my next blog post.