Tag: AI Builder

Posted on

Making Field Service Smarter using AI Builder

Quality audit is part of a work management life cycle in most Field Service organizations. To ensure customers are satisfied, equipment is up and running, and there are no SLA breaches, a quality audit is conducted in all or statistically sampled work orders. This often means creating another booking, or a task, or creating a follow-up work order for quality check and of course additional resource requirements – but is there a smarter solution possible in Dynamics 365 Field Service? This post explores the possibility of using AI Builder.

The Challenge:

In our Contoso field service organization, this is how the work happens:

  1. Field Technicians are booked to conduct the work
  2. Field Technicians complete the work, take a picture and uploads in the app
  3. Supervisors take a look at the picture, make a conscious call to either approve or visit a site to confirm work is properly completed
  4. Work Order is completed after the quality check has passed

For the point (3) above, we’ll use AI Builder’s object detection model to train it on pictures of quality vs faulty work, and then once a model is trained, hook it up with Field Service to check the quality of work and then change the status of Work Order automatically. This is how the process will look like:

The Setup:

We’ll need the following Power Platform products:

  1. Dynamics 365 Field Service
  2. AI Builder
  3. Power Automate Flow

All of the above can be setup as a trial for this post.

As for the images of quality vs faulty work, I’ve tapped into vast datasets available on Kaggle and found this one with the images of Lego bricks. For our experiment today, we’ll use 20 images each of 4×4 vs 1×2 bricks to differentiate quality vs fault work:

Faulty Work:

Quality Work:

The Play:

Following are three steps to run this play:

  1. Dynamics 365 Setup
  2. Build a model in AI Builder
  3. Create a flow to call AI Builder model with Dynamics 365

1. Dynamics 365 Setup:

Create a new field on Work Order entity:

Name: Completed Work

Type: File

….and drop it on a form.

Field Technicians upload the image of completed work in this field. We’ll use Power Automate to call AI Builder model on this field’s content.

2. Build a model in AI Builder

Create a new AI Builder model of type Object Detection and name it as ‘Quality Check’. Click Next..

Select ‘Common Object’ and click Next..

Add two ‘Object names’ as Quality and Faulty work. Click Next.

Click on Add images and upload all 40 images..

Click on Add images and upload all 40 images..

Click on Tag images and start tagging images one by one as Quality…

and Faulty work:

Let’s have a look at the Summary and then click on Train. It will take few minutes.

Once trained, run a few Quick Test. Seems to be working for me 🙂.

Quick Test 1:

Quick Test 2:

Create a flow to call AI Builder model with Dynamics 365

Let’s go to Power Automate flow and click New. We will use ‘Common Data Service (current environment)’ trigger and have the following actions:

Trigger: When a record is created, updated or deleted on Work Order

Steps:

  1. Get file from Work Order’s file field
  2. Predict action to call our AI model and pass on the image
  3. Based on the prediction, either mark Work Order as Closed or send an email to Field Technician

Here is our Power Automate flow:

..the second step:

the third and following steps…

All done! Start the play by uploading the image of completed work (in this case 1×2 or 4×4 lego brick images). The flow will run, send the image to AI model, and update the status of Work Order or send the email. Successful output:

Viola! That was fun and what we just experimented with is, in fact, a game changer in terms of making enterprise applications (like Dynamics 365 Field Service) really smarter – coupled with the fact that you can get this up and running in a very short timeframe and without expensive solution development/deployment.

Hope this was useful. If you have any suggestions/questions, please do reach out – would love to hear from you.

Posted on

Developing heart risk calculator using AI Builder

The idea of this article is to experiment with AI Builder to develop an app for self-assessment of heart-related diseases. An AI-powered based app could tap into historical data and identify patterns, and then use them to assess the user’s entered data to predict if there is a risk of heart diseases or not.

The app is not to be used for medical advice or prescription, rather this is an experiment to use Prediction model of AI Builder with a #nocode approach.

The first step to accomplish this was to find data on heart-related risks. Luckily I got it, thanks to Kaggle. You can go here (it is one of the many publicly available datasets) and download the data.

  1. Next, goto Power Apps maker portal, create a solution and create an entity for storing heart diseases statistics. Create a model-driven app and add the entity into the app.
  1. Import data into the entity
  1. With data imported into the system, click on AI Builder on the left. Click on Prediction
  1. A new window will open, give a name to your model
  1. Select entity and the field which holds prediction outcome
  1. Select fields that contribute to the outcome (for example, de-select created on, modified on etc. if they are selected)
  1. Next, you can filter the data to be analyzed. In this case, we just selected ‘Skip this step’ and clicked ‘Next’
  1. View the summary and click on Train
  1. This data will be then fed into training the model
  1. Once training completes, publish the model. Accuracy is pretty impressive.
  1. Once published, details screen will show (a) option to run model by clicking on ‘Use Model’, (b) percentage distribution of individual field’s contribution to the outcome, and (c) how model is being used (currently it automatically syncs the outcome with the database daily).

It is important to note here that unlike other AI models, Prediction models run daily. If you want to run it on demand, click on Use Model and then click on Run Now.

  1. Now lets’s go back to the app and create a new user profile.
  1. Now either wait for Prediction model to run or force it to run now (step 9 above).
  2. Once the model has run, go back to your record. Here is a snapshot of my data:

The model has predicted user does not have the risk of heart diseases, viola!

Now that I have seen this running, I can already look back and think of past implementations where this model through AI Builder can add a lot of value. What do you think? Please feel free to share your feedback on Twitter.

Happy AI Building!

Posted on

Building Crowd Counting Model using AI Builder

Counting the number of people in a photo or video feed is one of the interesting problems in the world of AI, and for a good reason because “number of people/given area” is a useful metric for number of practice purposes. For example:

  1. Retail: Knowing foot traffic in a specific store area or floor indicates how impactful the discount promotion is, or how popular a particular brand/product is
  2. LGAs/councils/city governments: Lot of smart cities want to know near real-time information on how many people are in a particular public space, park, or tourist spot to ensure adequate resources are available to ensure city services are functioning correctly
  3. Police/law enforcement/emergency services: Knowing the crowd size, helps emergency services to deploy resources appropriately

Also worth mentioning, like any other use of AI models, this is an area where functioning under the theme of AI for good is very important.

Data and tool

To experiment how AI Builder can help in crowd counting with #nocode approach, the first to-do is to secure data. Kaggle is one of the best resources for publicly available data. You can search and find a lot of data sets related to any topic. I found this one interesting and downloaded the data set on my laptop.

The tool is an AI Builder Object Detection model.

Let’s dive in…

  1. Goto Power Apps maker portal and click on AI Builder on the left. Click on Object Detection (it is under preview at the moment).
  1. A new window will open, give a name to your model
  1. You can select objects from CDS or your local machine.
  1. Select device..
  1. ..and add images (downloaded previously from Kaggle dataset). Images will upload
  1. Mention which object(s) you want AI model to auto-detect. In our case, this is only ‘Person’
  1. Next, we need to identify the object (Person – in our case) in each of the images. Click on each image..
  1. and tag all the persons
  1. AI Builder screen will show your progress on this tagging exercise
  1. When all the images are tagged, click on ‘Done tagging’
  1. Click Next
  1. This data will be then fed into training the model
  1. When the training will complete (may take a few minutes), it will show the degree of accuracy

It is not ideal but is not too bad for a #nocode experiment. Click on Publish. There are two ways to use this model; in a Power Automate Flow or in a Power Apps app. For this experiment, we will create a quick Canvas App. Click on Power Apps.

  1. This will open a familiar Power Apps maker portal. Add a button with the label ‘Analyze’ and a Data table.
  1. Behind a button, set output of the AI model to a ‘Persons’ collection
  1. Bind Data table with ‘Persons’ collection
  1. Save and experiment. Select image, let the AI model run and click on Analyze.

The model is returning the number of objects (persons) it identified with quite an impressive accuracy.

Please do let me know your feedback, always love to hear back from #PowerApps community. Happy Power-apping!

Posted on

Identify key words in Power Platform 2020 Release 1 plan using AI Builder

This is a part-fun and part-experimenting AI Builder article. Microsoft released 2020 Release Wave 1 Plan for Power Platform back in January. It is full of exciting stuff; all the new features, enhancements and updates planned circa April this year — and it is long read; 149 pages to be exact. Now, what if we could run some sort of ‘smart algo’ on this document to identify all the keywords (oft-repeated words) in it! This will give a new angle into what is the recurring theme, oft-repeated objectives and important buzz words that are mentioned in the release plan. Now before we go into details of what was that ‘smart also’ and how it was achieved, here is the results of the exercise:

Very insightful, right? And as you may have guessed, the ‘smart algo’ here is AI Builder. Let’s get our hands dirty…

  1. Goto Power Apps maker portal and click on AI Builder on the left.
  1. Click on Key Phrase Extraction model
  1. Click on ‘Use in a flow’
  2. This will lead you to Power Automate flow creation screen. Add a manual trigger and add ‘Text’ input
  1. A key limitation currently with Key Phrase Extraction model is that it supports the analysis of 5120 characters only, so we divided total length of the text with 5120 and then ran a loop to call Key Phrase Extraction model repeatedly, pass a substring of fewer than 5120 characters from the complete text and then save all the keywords in one variable. Add Predict action, select Key Phrase Extraction model and pass on a substring of complete text (less than 5120 characters). Add an action to send keywords using email
  1. Run the flow….
  1. ..and copy-paste Release Plan text and click on Run Flow
  1. Flow successfully runs, repeatedly calls Key Phrase Extraction AI model and sends all the identified keywords.

Note: The approach does return duplicates because in each iteration it processes 5000 characters, I had to de-duplicate result manually in excel but can’t see any reason why same can’t be done using Power Automate flow.

This was a quick demonstration of the power of Key Phrase Extraction AI model. In terms of its real-world usage, it could be used to review customer feedback, case descriptions, partners communications etc. to identify the theme or key elements of the conversations, which could be then addressed appropriately. Hope you find it useful, if you have suggestions or questions, please feel free to share.

Posted on

Automatically process Uber Receipts for expense approvals using AI Builder

A common scenario in most of the businesses is for employees to collect their receipts of the expenses made and then submit expenses along with receipts at the end of the month (or week/fortnight) for reimbursement. This creates a long queue (aka processing time and possible delay in processing) for HR/accounts team to review each receipt for approval. In this article, I have experimented with AI Builder to see if it can help in making the process faster and easier.

An example scenario here considers submission of an expense through a model-driven Power Apps app, but this could be used in Canvas Power App, Dynamics HR, emails or even any other system that connects with CDS. The process is to auto-approves the expense in expense in record tallies with the ‘Total’ field in Uber receipt or sends for further approval (manual offline process).

This demonstration requires three components to be built:

  1. A model-driven Power Apps app with an expense entity and form
  2. An AI Builder model to process Uber receipts
  3. A Power Automate flow to run on the create/update of expense, trigger AI Builder model and based on the result, change expense status

Step 1. Model-driven Power Apps app
Create a new PowerApps model-driven app and create an entity ‘Travel Expense’ entity with three fields:
a. Title
b. Total Expense ($ value)
c. Status

Note: An entity should support notes and attachments.
Also note: this step is not discussed in detail in this app. If model-driven apps are new for you, have a look at this excellent Microsoft docs resource.

Step 2. Create an AI Builder Model
Goto Power Apps maker portal and click on AI Builder on the left.

Click on Form Processing. You’ll notice the Examples and Best practices tab which have great tips on setting up AI models. Enter the name of your model (Read Uber Receipts – for example) and click on Create.

Next screen will ask you to upload documents. Select the files and upload.

Once the documents are uploaded, click on Analyze.

Now the AI engine will run..

..and it will ask you to tag the fields which it was able to find in the documents.

Select the Total field on the document and label it.

Click ‘Done’ and click ‘Next’

Next click on Train to train the newly created model

Once the training is completed..

..go to the Details page. This shows the document that is used to train the AI model, field identified during form processing and how to use the model (Power Automate and Power Apps).

You can also do the quick test by uploading another document and let the AI model find data in the document. Publish the model.

Step 3. Create Power Automate Flow

Go to the solution (created in Step 1) and click on New-> Flow to create a new flow.

Create a flow with a CDS trigger on the entity that we created in Step 1. Next, find notes associated with the expense record, fetch attachment in Notes and then use ‘Apply to each’ to run an AI model on each file. For calling the AI model, use ‘Predict’ action.

Next, fetch the result from the AI model and compare with the expense entered by the user in the form. If both match, then approve the record, else send for manual review.

Lastly, let’s test it out:

  1. Create a new expense record and attached an Uber receipt:

.. in the background, the flow ran:

Record gets updated to manual approval process (In Progress) if expense record total doesn’t match total in receipt:

..otherwise it gets auto-approved:

Hope you will find this useful. Please feel free to share your feedback 🙂