react-cs-images-suggestion: a SPFX sample webpart using Azure Cognitive Services

*Update 12/03/2019*
This article was mentioned in SharePoint Dev Weekly – Episode 30 thank you Vesa and SharePoint Team! I made a mistake, this is not AI (Artificial Intelligence): this is just Cognitive Services use: that’s true 😀 ! When I started to think about this sample, I was thinking to use Luis or other services provided by Azure AI with training and deep learning. In the end, I used Azure Cognitive Services, so I renamed react-ai-images-suggestion in react-cs-images-suggestion .
Thanks again for the mention, #sharingiscaring 😉

Hi guys,
these days I was working on a new contribution, I would liked to realize an “image suggestion webpart” like Bing Visual Search , ovviously working on SharePoint Online and SharePoint Picture Library. So I started to search if someone did something similar and I found a great article (I would like to thanks rasper87 for the inspiration in steps 1-4 of the architecture) talking about auto-tagging images using Azure Cognitive Services and Microsoft Flow.

react-cs-images-suggestion in action!

The results of my work is a little SPFX webpart using the open source library react-dropzone to upload an image and immediately search for visually similar images inside a SharePoint Online Picture Library, by the employment of Azure Cognitive Services – Computer Vision and SharePoint Search REST API.
Keywords of the images in the library are enriched during upload thanks to a Microsoft Flow connected to the Azure Cognitive Services Computer Vision.

Architecture

Here you can find the full architecture

Configuration setup: SharePoint, Azure Cognitive Services and Flow

First of all, you need to create

• A SharePoint modern team site on your O365 tenant.
• A Picture Library (name it AI Picture Library if you want, following screen use it), inside your new site
• An Azure Cognitive Services instance on your Azure Subscription

That’s perfect! Everything is ready for Flow Configuration, let’s start!

And here we are, let’s test the first part of the solution 🙂

Code focus

Full code solution of react-cs-images-suggestion is avaliable on my GitHub profile but I would like to focus on two snippets

How to upload a file with a post to Azure Cognitive Services in SPFX

private async _getTagsForImage(fileInfo: any): Promise<string[]> {
    const httpOptions: IHttpClientOptions = this._prepareHttpOptionsForVisionApi(fileInfo);
    const cognitiveResponse: HttpClientResponse = await this.props.context.httpClient.post(this.cognitiveServicesVisionUrl, HttpClient.configurations.v1, httpOptions);
    
    const cognitiveResponseJSON: any = await cognitiveResponse.json();
    const tags: any = cognitiveResponseJSON.tags;
    return tags;
}

private _prepareHttpOptionsForVisionApi(fileInfo: any): IHttpClientOptions {
    const reader = new FileReader();
    reader.onload = (event) => {
      //console.log(reader.result);
    };

    reader.readAsDataURL(fileInfo);
    const httpOptions: IHttpClientOptions = {
      body: fileInfo,
      headers: this._prepareHeadersForVisionApi(),
    };

   return httpOptions;
}



private _prepareHeadersForVisionApi(): Headers {
    const requestHeaders: Headers = new Headers();
    requestHeaders.append('Content-Type', 'application/octet-stream');
    requestHeaders.append('Cache-Control', 'no-cache');
    requestHeaders.append('Ocp-Apim-Subscription-Key', this.cognitiveServicesKey);
    return requestHeaders;
}

How to call SharePoint Search REST API using pnp in spfx

import { sp, SearchQueryBuilder, SearchQuery } from "@pnp/sp";

//cut 

var q = String.Format('and(or({0}),filetype:"jpg")', tagsQuery);
const _searchQuerySettings: SearchQuery = {
          TrimDuplicates: true,
          EnableFQL: true,
          RowLimit: 50,
          SelectProperties: ["Title", "SPWebUrl", "DefaultEncodingURL", "HitHighlightedSummary", "KeywordsOWSMTXT"]

        }

        let query = SearchQueryBuilder(q, _searchQuerySettings);
        sp.search(query).then((r: any) => {//it was SearchResults
          console.log(r.ElapsedTime);
          console.log(r.RowCount);
          console.log(r.PrimarySearchResults);
          const data = [];
          let _data = {};
          r.PrimarySearchResults.forEach(element => {
            console.log(element);
            _data = {
              key: element.UniqueId,
              name: 'Item ' + element.Title,
              tags: element.KeywordsOWSMTXT,
              thumbnail: element.DefaultEncodingURL
            };

            data.push(_data);
          });

Hope you enjoy my new code contribution!

Cheers
Federico