Data virtualisation using the ISupportIncrementalLoading interface

Sometimes the data set an app works with is so large that it shouldn’t be stored in memory. For example, if your photo collection contains tens of thousands of photos it is not sensible to load all the photo thumbnails for display when the page loads. This is because it will require a substantial amount of memory to store the data that’s being displayed. Generally, high memory use degrades the experience for all apps on the system. In addition, the chances of a suspended app being terminated rises with the amount of memory being used by the active app. This is also true for the chances of the app being terminated when it’s inactive. In such circumstances you should implement a data virtualization strategy, where you load an initial portion of the data set into memory, and then load further data incrementally on-demand.

Incremental data virtualization is one such strategy and sequentially retrieves data. For example, a GridView that uses incremental data virtualization and contains 10,000 items could choose to retrieve only the first 20. Then the next 20 items can be retrieved as the user pans through the grid. Each time more items are retrieved the scroll bar for the grid becomes smaller. To implement this type of data virtualization you must use a collection that implements the ISupportIncrementalLoading interface. This interface defines one method and a property:

• HasMoreItems is used to determine if the collection has more items available to be retrieved.
• LoadMoreItemsAsync is invoked every time the control that binds to the collection needs to retrieve more items for display.

By implementing this interface, if data is being displayed in a GridView, when the scroll thumb reaches the edge of the screen, after having checked the HasMoreItems property, the LoadMoreItemsAsync method is invoked to retrieve more data for display.

Implementation

In order to implement incremental loading we must have a class that inherits from the ObservableCollection<T> class, and which implements the ISupportIncrementalLoading interface. The IncrementalLoadingPhotoCollection does this.

public class IncrementalLoadingPhotoCollection : ObservableCollection<IPhoto>, ISupportIncrementalLoading

{

    public bool HasMoreItems { get; private set; }

    public IncrementalLoadingPhotoCollection(IRepository repository)

    {

        ...

        HasMoreItems = true;

    }

    public IAsyncOperation<LoadMoreItemsResult> LoadMoreItemsAsync(uint count)

    {

        return InnerLoadMoreItemsAsync(count).AsAyncOperation();

    }

}

The IncrementalLoadingPhotoCollection constructor sets the HasMoreItems property to true, indicating that there is data which can be incrementally loaded. The LoadMoreItemsAsync method will be automatically invoked to load data for display on the MainPage. It does this by invoking the InnerLoadMoreItemsAsync method.

private async Task<LoadMoreItemsResult> InnerLoadMoreItemsAsync(uint expectedCount)

{

    var actualCount = 0;

    ObservableCollection<IPhoto> photos;

    try

    {

        photos = await _repository.GetPhotosAsync(_photoStartIndex);

    }

    catch (Exception)

    {

        HasMoreItems = false;

        throw;

    }

    if (photos != null && photos.Any())

    {

        foreach (var photo in photos)

        {

            Add(photo);

        }

        actualCount += photos.Count;

        _photoStartIndex += (uint)actualCount;

    }

    else

    {

        HasMoreItems = false;

    }

    return new LoadMoreItemsResult

    {

        Count = (uint)actualCount

    };

}

This method creates a new ObservableCollection of IPhoto and calls the GetPhotosAsync method of the FileSystemRepository class to retrieve the first 20 photo thumbnails. Provided that data has been retrieved, it’s then added to the IncrementalLoadingPhotoCollection for binding to by the MainPage view. When there’s no more data to retrieve an exception occurs and the HasMoreItems property is then set to false, indicating that there’s no more data to retrieve. A LoadMoreItemsResult structure is returned, which wraps the asynchronous results of the LoadMoreItemsAsync call. This structure contains a Count field, which indicates the number of items that were loaded.

The GetPhotosAsync method in the FileSystemRepository class is responsible for retrieving up to 20 thumbnails on each invocation.

public async Task<ObservableCollection<IPhoto>> GetPhotosAsync(uint startIndex)

{

    ...

    var files = await fif.GetFilesAsync(startIndex, _defaultPageSize);

    ...

}

The first time GetPhotosAsync is called it retrieves the thumbnails for the first 20 photos on the file system (index 0 to 19) by calling one of the overloads of the GetFilesAsync method. The _defaultPageSize field is set to 20, indicating that a maximum of 20 FileInformation objects will be returned. Therefore, the next time the GetPhotosAsync method is called it will retrieve the next 20 photo thumbnails (index 20 to 39), and so on. The starting index for each call to GetPhotosAsync is stored in the _photoStartIndex field in the IncrementalLoadingPhotoCollection class.

The GridView on the MainPage binds to the Photos collection in the MainPageViewModel class. The Photos collection is of type IncrementalLoadingPhotoCollection.

public IncrementalLoadingPhotoCollection Photos

{

    get { return _photos; }

    private set { this.SetProperty(ref _photos, value); }

}

When the MainPage is navigated to, the OnNavigatedTo method in the MainPageViewModel class will be executed. This simply initializes the Photos property to a new collection of type IncrementalLoadingPhotoCollection.

public override void OnNavigatedTo(object navigationParameter, NavigationMode navigationMode, Dictionary<string,object> viewModelState)

{

    base.OnNavigatedTo(navigationParameter, navigationMode, viewModelState);

    Photos = new IncrementalLoadingPhotoCollection(_repository);

}

The overall effect is that when the MainPage is loaded a subset of your photo thumbnails are retrieved from the file system for display in a GridView. When you you pan through the GridView more photo thumbnails are incrementally retrieved from the file system on-demand for display. This avoids having to store all the photo thumbnails in memory, thus decreasing the memory footprint of the app and increasing the experience for all apps on the system.

Summary

This blog post has extended the sample app by implementing data virtualization using the incremental loading strategy. This was achieved by creating the IncrementalLoadingPhotoCollection class, which implements the ISupportIncrementalLoading interface. This class uses the FileSystemRepository class to retrieve 20 photo thumbnails at a time, and incrementally retrieves further thumbnails on-demand as the user pans through the GridView that displays the thumbnails. This avoids having to store all the photo thumbnails in memory, thus decreasing the memory footprint of the app and increasing the experience for all apps on the system.

The sample code can be downloaded here.