Articles, Power Automate
by anjcerbolles

AI Builder Form Processing Ideas

We’re in the Digital Transformation age and everyone are using mobile phones, computers, and tablet to solve business problems. We now automate business processes to work productively and efficiently. One of the main process that business users wanted to automate is the paper approval. Business users want to automate this because of the issue where you need to look for the approver in his/her desk to approve your form. If the approver is out of reach, your paper will just be in your desk waiting until the Approver arrives.

With all the digital transformation and automation, Power Platform released General Availability of AI Builder last October 2019 but some of its features are still in preview. And one of the features that I’m playing around is the Form-Processing (Preview).

Scenario

While I’m doing some proposal for work, I did this scene where the user will create a list in SharePoint, attached an invoice form to the item and then submit. Once submitted, the Power Automate will trigger to get the attached file from the newly created item. It will then read the invoice form using AI Model Form-Processing. Once it is extracted, the data from the invoice form using the AI Model will then update the list item in SharePoint. The below image is the Power Automate that I’ve been working on. I will show you the detailed setup that I have for my Power Automate in my next post.

For now, below is the image for the flow.

Power Automate Home Action items My flows Create Templates Connectors Data Al Builder Solutions Learn FormProcessing Search for helpful resources When an item is created Get attachments from current item Apply to each Environments Personal (default) (orqfa Save te_9 Flow checker Test Select an Output from previous steps body x Get file content in current item Predict Parse 'SON Update item Add an action + New step

Flow for picking up the attached file from SharePoint item

How Does it Work

Form-Processing extracts data fields and tables from the form. In your Power Apps environment, you can see the AI Builder in the left pane of the Web Studio. From there, you can build your Models like Form Processing, Object Detection, Prediction and other available AI productivity.

AI Builder in Power Apps

In building the  AI Model using the Form-Processing, you need to have a minimum of 5 documents to train your model. The model can detect duplicate document. Since it is a duplicate, it will ignore and will not upload the document.

Information if the document uploaded has a duplicate.

After you are done uploading all the documents, you start to Analyze the document.

Add documents 5 documents added Select form fields Model summary -F Add documents Get sample documents Add document examples Form? Analyzing Were analyzing your documents This may take a few minutes 5 documents added

Analyzing the document uploaded

After analyzing all the documents, now you start to select the fields that you are going to extract from the document.

Select all X Clear all 3 form fields selected INVOICE V Done FormProcessingAlModel form Selected fields All fields Bill To table_O (Item, Quantity, Rate, Tax (2%) ofl

Selecting the fields that you want to extract from the document

Once done selecting the fields, you train the model.

Your model is training This may take a while. You can close this window and come back later. Go to models

Train the model

After training the model, you will then publish it. After publishing the model, you can now start using it in your Power Automate and Power Apps.

[3 Power Automate Home Action items My flows Create Templates Connectors Data Al Builder Build Models Solutions Learn P Search for helpful resources Share (preview) Settings Your model isn't published yet. Publish to use it in apps and flows. Models > FormProcessingAlModel Form Processing (preview) • Not published • Anj Cerbolles Selected fields Bill To Tax (2%) table_O See more Environments Personal (default) (orqfa How your model is used Power Automate Model isn't published Power Apps Model isn't published Training document Publish See details Quick test

Your Form-Processing Model

My Thoughts

This is only my own experience and comments with regards to AI Builder Form-Processing. As I went through my testing and configuration, some things need to address and this would be useful as well. As I have some experience with AI application for the extraction of the data through scanning of the documents. Probably this would be helpful as well for this feature.

  1. When you are done uploading the file and done with analyzing the document if you don’t want to use the document that has been analyzed and need to upload a new document. The new document will not be analyzed anymore. You need to start creating a new model again and start analyzing.
  2. There is no option for the trained model to manually re-train it. Let’s say for example you have more than 20 documents that already analyze but then, the document doesn’t properly extract the fields or the tables. You can’t re-train the model.
  3. There is no option for you to split the documents if there are multiple pages. I know part of the best practice is no multiple pages. If these can be extended to multiple pages, this could be a great value to the app. Especially when the form reaches even the two pages only.
  4. You can’t adjust the positioning of the extracted fields or tables. Let’s say, you have a four-column table. S/No, Description, Rates, Remarks are the headers. During the analysis of your model, extracted fields of the Rates are somewhat nearer to the fields of the Remarks. If the table of the document that you are going to extract is not the same alignment for the trained model. There will be a possibility that the Rates will be read as Remarks if the Description field is wider than the trained model.
  5. If there could be a manual training of the model, we can define the unique keyword of the document. From there, you can set manually the coordinates of the keywords. You can identify if the keyword that you are looking are either in the TOP header or in the footer. 

We know that Form-Processing is still in Preview. There is still room for improvement. If this thing can be addressed, AI Builder Form-Processing can be a powerful tool for extracting forms. And because it is under the Power Platform, you can leverage the Power Apps and Power Automate more and it would be easy to build and can leverage more on the business process.

Featured image by Pixabay