Power Platform : How to extract Form document data using AI Builder and Microsoft Flow

By | July 10, 2019

Introduction

In our day to day business processes, we can use Artificial Intelligence to advance business operations. The AI Builder (preview feature) is the new Power Platform feature that allows us to use Artificial Intelligence to easily automate our business processes and predict outcomes. AI Builder currently supports four types of AI model, Form processing model, Text classification model, Binary classification model, and Object detection model. Each model serves a different purpose, we can choose the type of model based on our requirement.

In this blog, we will illustrate how to create and use Form processing model with the help of a real-world business scenario.

What is Form processing model?

Form processing model as the name suggest, processes the forms or documents that have data in key-value pair. It extracts the document data and makes it available for further use. It uses the existing sample documents to analyze the new documents and extract information. The accuracy of outcome of this model depends on the sample documents that we used to learn the model.

Scenario:

A renowned construction company uses CRM to manage their business process. For every new work the sales person creates a new Project entity record in CRM. Contractors (Not CRM users) assigned to that project, analyze the work and prepare a quote and that quote get attached to the project record as notes attachment. Sales person then manually downloads the attachment, reads the quote and updates the quote details on Project entity. They have to do this every time when any new quote document is attached to the record.

This process is quite time consuming and requires human intervention to get the most recent quote information on the project.

Here, we can make use of Form processing AI model to automate this process. Given below are the steps to automate this process using AI Builder.

1. Enable AI Builder: Make sure that AI builder is available and enabled for the environment. Sign in to Power Platform Admin center. In the admin center go to Environments -> <select environment> -> Settings -> Features.

In Features page check if Create AI models in PowerApps is enabled.

Form Processing AI Model Microsoft Flow Power Apps

2.Create Form Processing AI Model: Sign in to https://web.powerapps.com. Go to ‘AI Builder’ -> Build -> Form processing.

Form Processing AI Model Microsoft Flow Power Apps

This will open the below screen. Give name for the model. We can see this model requires sample documents. We can see the examples of document. Click on ‘Examples’ to see the example of sample document.

Form Processing AI Model Microsoft Flow Power Apps

3. Prepare sample documents: Go through Form processing AI model requirements and limitations. The documents that you have must meet these requirements otherwise model will fail with the errors. So if you ever face error while analysing the model make sure that your sample documents meet all the requirements. As an example we have taken below Quote report (PDF format) which the contractor fills up and uploads against the project.

Form-Processing-AI-Model-Microsoft-Flow-Power-Apps

We have 5 different versions (data) of the same report. We will use these five pdf files as a sample data for the Form processing model.

4. Analyze Documents: Click on ‘Add Documents’ button, select all the sample pdf files and upload it.

Form Processing AI Model Microsoft Flow Power Apps

Once all the documents are uploaded close the window and click on ‘Analyze’ button. This will save the model and start analyzing the documents.

Form Processing AI Model Microsoft Flow Power Apps

This process will take few minutes and will ask you to select a field. Analyze steps, analyze the structure of the document, identifies the key value pair, table data and expose are fields that we need to select in the next step.

5. Select form fields: Here we need to select data that we want to extract from the document. Click on ‘Process Quote Model form’, this will open the below screen with all possible data that the Model can extract.

Form Processing AI Model Microsoft Flow Power Apps

We can either select all the fields or we can select those fields that are needed. Here we have selected the Total Amount and Prepared By field. Click on ‘Done’ button to save the field selection for model. Click on ‘Next’ button to train the model using uploaded sample documents.

6. Train, Test and Publish Model: Click on ‘Train’ button to train the Form processing model.

Form Processing AI Model Microsoft Flow Power Apps

Below screen shows model training is completed. Click on ‘Go to the details page’ button. This will take us on the Model page.

Form Processing AI Model Microsoft Flow Power Apps

You can also navigate to the model details page by clicking on ‘Model’ -> <click on Model from the list of model>

Form Processing AI Model Microsoft Flow Power Apps

We can test the model by using ‘Quick test’ button. Upload the new document and check if it extracts the information that we wanted or not. Finally click on Publish button to publish the model.

7. Use Form processing Model to automate business process using Microsoft Flow:

Create a new solution in Microsoft flow and then create a new Flow under that solution and setup it to trigger on creation of notes for Project entity. Here we will consider Opportunity as the Project entity. We have ‘Quotation information’ section on Opportunity/Project entity which has two fields ‘Total Amount’ and ‘Prepared By’ field.

Our flow will trigger on any upload of notes attachment on project entity, then it will call and pass the uploaded document to Form Processing Model and finally, will read and update the response returned by Model on opportunity fields.

Add a ‘Predict – Common Data Service (Current environment)’ action and select our newly created Form Processing Model.

Form Processing AI Model Microsoft Flow Power Apps

In request payload pass JSON string in below format;

Form Processing AI Model Microsoft Flow Power Apps

{

“base64Encoded”: “EXPRESSION”,

“mimeType”: “image/jpeg”

}

This expression must return document content in base64 format. In case of CRM notes document, we can directly pass Document field. The expression must be enclosed by base64() based on which connector the files comes from. We can have SharePoint connector or any other connector which will give files to be passed to AI Model. The Mime type also depends on the type of document. If the document type is pdf then pass ‘application/pdf’ and if document type is image then pass correct mime type ‘image/jpeg’

Now save the Flow and test it by clicking on ‘Test’ button. Check the logs of Flow and copy the result of Predict action and past it in code editor i.e. either Visual Studio or Visual Code.

Form Processing AI Model Microsoft Flow Power Apps

Remove “\” characters from output that we copied from Predict action.

The output will have result for the fields that we selected while creating a Model. Here you can see the value for Total Amount and Prepared By.

Form Processing AI Model Microsoft Flow Power Apps

Add Parse JSON action, select ‘Response Payload’ in the content and paste the JSON output copied in code editor (JSON string after removing “\” character) in the Schema.

Form Processing AI Model Microsoft Flow Power Apps

Now we are ready to use the response of Form Processing model to update the quote details in CRM. Add ‘Update record’ action of Common data service and use the values from Parse JSON action.

Form Processing AI Model Microsoft Flow Power Apps

body(‘Parse_JSON’)?[‘predictionOutput’]?[‘labels’]?[‘Prepared by ‘]?[‘value’]

body(‘Parse_JSON’)?[‘predictionOutput’]?[‘labels’]?[‘Total Amount’]?[‘value’]

Note: To use AI Builder Models in a flow we must create flow in solution.

Conclusion

In this way, we can use Form Processing AI model of AI builder to process the documents, analyze it and extract data from it. Based on requirement we can adjust the Microsoft Flow to when it should trigger, call AI model to get data from the document and use that data.

70% of global 2000 companies apply gamification to improve productivity and returns!

Gamifics365 – Spin the magic of games within Microsoft Dynamics 365 CRM to improve user adoption, enhance productivity, and achieve company goals!

14 thoughts on “Power Platform : How to extract Form document data using AI Builder and Microsoft Flow

  1. Laura

    Hi! This article is super helpful in what my company wants to accomplish but when creating my flow I have no option for “Predict” Do you know why this is?

    Thanks!!

    1. inogic

      You have to make sure that you are in the right environment. Change the Flow environment and try again. There is an option to select the environment below the sign out section.

      Thanks!

      1. Matthew McClelland

        I’m having the same issue as Laura, what environment should I be looking for? I only have one available, unless I create my own trial environment.

        Thanks

        1. Inogic

          Make sure you are not using the Default environment. And most importantly, this feature is only available for flows that you create from within the context of a solution.
          You need to create a solution first.
          From within the solution choose new flow
          you should now be able to access the Predict action.

          HTH

  2. Paul

    Hi
    A great article. heres a bit of a paranoid question…The image I am using to create/train model and then also the images I then use to test the model. Where is that image nformation stored ?
    ie. I may want to create a model that reads in details from an invoice(s), Is that invoice infomation and images being saved/stored anywhere outside of my model ?
    Thanks
    Paul

    1. inogic

      Very good question. All the AI builder configuration get stored in Common data service. You can do an advanced find in CRM and check for AI related entities for example, AI configuration, AI Model, AI Form processing document, Text Classification model.

      The images that you use to train/create model get stored in AI Form Processing document entity.

      Thanks!

  3. Ted

    I have the same problem as Laura. The Predict option is not available via Flow. The only CDS options I see are related to Records (create, get, list, update and delete). In the upper right, the only environment I see is my current work environment.

    1. inogic

      Hi,

      In order to use Predict Common data service (Current Environment) flow connector, you must create the flow in solution. So first, create a solution and then create a new flow inside it. After this, you will be able to see ‘Predict common data service (Current environment)’ connector. create-flow-solution

      Thanks!

  4. Gerald Gjini

    Is there any way to get the parsed the JSON into a CSV/TXT/Excel instead of CDS?

    1. inogic

      Yes, there is a way in which you can parsed the JSON into CSV/TXT/Excel. For this, you would need to call action for Excel Online connector and write the JSON response from AI builder into excel.

      Hope this helps.

      Thanks!

  5. Jenn Fres

    What if I have multiple invoices I need to extract from? Different format from each other. Can I have just 1 flow for the different models? How do I go about that?

    1. Inogic

      Hi Jenn,

      I am not sure if I have understood your requirement correctly. I am assuming you have invoices from different providers and have different layouts and want to extract information from those. Currently, we can build AI Form processing model for the files with having same layout. For example, invoices with same layout, but each uploaded invoice is from a different month. (Refer this link.) If there are going to be fix number of different layouts (2 or 3) then you can go ahead and create AI model for each layout and can use those in Microsoft Power Automate.

      You can also read the conversation around the same requirement, https://powerusers.microsoft.com/t5/General-Power-Automate/AI-Builder-Forms-Processing-multiple-layouts/td-p/453174

  6. Talk TTIvein

    Thanks for wonderful article, I have created Flow and would like to trigger on case creation. If any document is attached to Case, i would like to process the document,extract information and store in another entity. I have created Predict model based on as many as 20 documents and trained the model. Whilst creating Flow how can i pass document in Document entry of Predict action. I have manually set Document type as application/pdf but not sure how to set document. Can’t see anything in expression.

    1. Inogic

      You need to pass the document in ‘base64’. If you are using notes attachment then you can directly pass the attachment of note. Retrieve notes of the case and then pass the Document.

      form-processing

      form-processing

      Thanks!

Comments are closed.