Reading Time: 3 minutes

*Update 12/03/2019*
This article was mentioned in SharePoint Dev Weekly – Episode 30 thank you Vesa and SharePoint Team! I made a mistake, this is not AI (Artificial Intelligence): this is just Cognitive Services use: that’s true πŸ˜€ ! When I started to think about this sample, I was thinking to use Luis or other services provided by Azure AI with training and deep learning. In the end, I used Azure Cognitive Services, so I renamed react-ai-images-suggestion in react-cs-images-suggestion .
Thanks again for the mention, #sharingiscaring πŸ˜‰

Hi guys,
these days I was working on a new contribution, I would liked to realize an “image suggestion webpart” like Bing Visual Search , ovviously working on SharePoint Online and SharePoint Picture Library. So I started to search if someone did something similar and I found a great article (I would like to thanks rasper87 for the inspiration in steps 1-4 of the architecture) talking about auto-tagging images using Azure Cognitive Services and Microsoft Flow.

react-cs-images-suggestion in action!

The results of my work is a little SPFX webpart using the open source library react-dropzone to upload an image and immediately search for visually similar images inside a SharePoint Online Picture Library, by the employment of Azure Cognitive Services – Computer Vision and SharePoint Search REST API.
Keywords of the images in the library are enriched during upload thanks to a Microsoft Flow connected to the Azure Cognitive Services Computer Vision.


Here you can find the full architecture

Configuration setup: SharePoint, Azure Cognitive Services and Flow

First of all, you need to create

  • A SharePoint modern team site on your O365 tenant.
  • A Picture Library (name it AI Picture Library if you want, following screen use it), inside your new site
  • An Azure Cognitive Services instance on your Azure Subscription

That’s perfect! Everything is ready for Flow Configuration, let’s start!

And here we are, let’s test the first part of the solution πŸ™‚

Code focus

Full code solution of react-cs-images-suggestion is avaliable on my GitHub profile but I would like to focus on two snippets

How to upload a file with a post to Azure Cognitive Services in SPFX

private async _getTagsForImage(fileInfo: any): Promise<string[]> {
    const httpOptions: IHttpClientOptions = this._prepareHttpOptionsForVisionApi(fileInfo);
    const cognitiveResponse: HttpClientResponse = await, HttpClient.configurations.v1, httpOptions);
    const cognitiveResponseJSON: any = await cognitiveResponse.json();
    const tags: any = cognitiveResponseJSON.tags;
    return tags;

private _prepareHttpOptionsForVisionApi(fileInfo: any): IHttpClientOptions {
    const reader = new FileReader();
    reader.onload = (event) => {

    const httpOptions: IHttpClientOptions = {
      body: fileInfo,
      headers: this._prepareHeadersForVisionApi(),

   return httpOptions;

private _prepareHeadersForVisionApi(): Headers {
    const requestHeaders: Headers = new Headers();
    requestHeaders.append('Content-Type', 'application/octet-stream');
    requestHeaders.append('Cache-Control', 'no-cache');
    requestHeaders.append('Ocp-Apim-Subscription-Key', this.cognitiveServicesKey);
    return requestHeaders;

How to call SharePoint Search REST API using pnp in spfx

import { sp, SearchQueryBuilder, SearchQuery } from "@pnp/sp";


var q = String.Format('and(or({0}),filetype:"jpg")', tagsQuery);
const _searchQuerySettings: SearchQuery = {
          TrimDuplicates: true,
          EnableFQL: true,
          RowLimit: 50,
          SelectProperties: ["Title", "SPWebUrl", "DefaultEncodingURL", "HitHighlightedSummary", "KeywordsOWSMTXT"]


        let query = SearchQueryBuilder(q, _searchQuerySettings); any) => {//it was SearchResults
          const data = [];
          let _data = {};
          r.PrimarySearchResults.forEach(element => {
            _data = {
              key: element.UniqueId,
              name: 'Item ' + element.Title,
              tags: element.KeywordsOWSMTXT,
              thumbnail: element.DefaultEncodingURL


Hope you enjoy my new code contribution!


2 thoughts on “react-cs-images-suggestion: a SPFX sample webpart using Azure Cognitive Services”

Leave a Reply

Your email address will not be published. Required fields are marked *