Sonoma Partners Microsoft CRM and Salesforce Blog

Predictive Vision Service - Step by Step

Today's blog post was written by Mike Sherry, Developer at Sonoma Partners.

Part 1: Creating and Using a Model

Salesforce recently rolled out a really cool feature for its new Einstein AI platform: Predictive Vision.

The Predictive Vision service allows users to upload pictures to a pre-trained model, and the model returns its best guess (plus a few other probable options) of what is in the picture.

The application of this service maybe isn’t immediately obvious, so let’s imagine you’re a mechanic and you have your own repair shop. Actually - let’s imagine you’re a starship mechanic. That’s more fun. Your clients are sending you pictures of their busted up ships; they want you to send them repair estimates, and – being starship pilots – they want them fast!

Ship's Broke...

Ship's Broke...

The galaxy is a big place, filled with lots of different starships, so you can lose a lot of time figuring out which ship you’re looking at. Predictive Vision to the rescue! Using Salesforce’s API, we can build a robust model of all of the different types of ships that you work on. Then, when a customer wants an estimate faster than the speed of light, all you need to do is run their picture through your model, and you quickly know what kind of ship you’re working with. Since you already have a Salesforce platform complete with replacement part prices, cost of labor estimates, and everything else that goes into your business, we can build a slick tool to tie it all together – but that will have to wait for another blog post. For now, let’s get up and running with Predictive Vision.

Let’s Get Started

It’s easy to sign up for the Predictive Vision service and hook it up to your Salesforce org. The documentation provides excellent step-by-step instructions for creating a certificate from your Salesforce org, downloading the certificate, and then using it to create a new account, so we won’t go over it in depth here. There are also code repositories provided by Salesforce that allow you to quickly create a basic Visualforce page that utilizes an already created image classifier in your Salesforce org.  All you have to do is clone the code repositories, copy the code into your org, and then follow the instructions provided to get the Visualforce page set up.

Now that we’ve got our very own account and it’s hooked up to our Salesforce org, we can try out the Predictive Vision service using the default image classifier. Since we need it to identify space ships, let’s see how the default classifier does…

It IS a Space Shuttle. Not Bad!

It IS a Space Shuttle. Not Bad!

The model is 99.9975% sure that this is a picture of a space shuttle. Pretty cool! Maybe we can use the default model and be done with this project right now! Let’s test it with the picture of our customer’s broken ship first.

Definitely Not a Can Opener

Definitely Not a Can Opener

Syringe… nope. Can opener… nope. Slide rule… nope. Space shuttle? Eh. Close, but still not what we need.

The image classifications are only as good as the model we use (and the model is only as good as the data we give it – more on that later). The default model is pretty impressive, but we’ll get much better results if we have a model specifically tuned to the kinds of images we want to classify. So, let’s see how to create a Model of our own.

Create a Dataset

To create a brand new model, we now have to leave the Salesforce org and work directly with our account. has a pretty simple REST API that allows you to create new datasets, add labels, upload photos, create models, and classify new images using the models you create. Their documentation shows what the REST calls look like if you’re using cURL to communicate with, but they’re pretty easy to translate into whatever language or tool you’re most comfortable with. There has been some talk from Salesforce that they will be releasing wrapper functions for Java, Node.js, and other languages in the future. For now, though, we have to make our own.

The first thing to think about when you’re working with the API is how you’re going to get an authorization token. There is a shell script available here which will generate a token for you. It’s designed to run in a Linux or Mac environment, but we were able to get it working in Windows with a little bit of editing.

The easiest solution, however, might be to simply load the sample Visualforce page that Salesforce provides us. They went through the trouble to write an Apex class that gets an access token, so we might as well use it! Remember when we tested the general model? Right below the image, there’s a field called “Access Token.” We can simply load that page and copy the token. It’s valid for one hour and if it expires, we can just reload the page to get a new one.

Sample Visualforce Page with an Access Token

Sample Visualforce Page with an Access Token

Here’s an example of a wrapper function we wrote using Node.js and its request library to create a new Dataset:

Mike sherry 5

To call the function, we just give it a name for the dataset and a list of the labels we want. In our case, the dataset will be called “Starships,” and we’ll give a label for each of the different models of starship we repair at our shop.

Mike sherry 6

The API call returns a JSON object with a dataset id and a list of all of the labels in the dataset, as well as ids for each label. Keep this information around, because we’re going to need it later.

Add Some Images to the Dataset

Now that we have a dataset, we need some sample images to help our model know what it’s looking for. Salesforce recommends providing at least 1000 images per labels, and they also recommend ensuring that each label has roughly the same number of pictures.

For this example, we used about 30 pictures per label and it gives pretty good predictions, so your results may vary if you’re using more or less labels and what sort of images you’re using as input.

To add an image to a dataset’s label, use the “Create an Example” endpoint. You’ll need to convert the image on your hard drive into a file stream. In the Node.js example below, we used the fs library to do the conversion. You’ll also need to pass in the model and label id so that knows where to put the image. You can send a request to the “Get a Dataset” or “Get All Datasets” endpoints if you forgot the dataset/label ids.

Mike sherry 7

Once you upload an image, it can take a few minutes for your to process it and have it ready to do more work (training, returning information about your model). You’ll know it’s ready when you can hit the “Get an Example” endpoint and it returns information instead of an error message. If you get an error, give it a few minutes and try again.

Train the Model

Once you’ve created a dataset and loaded all the images, training a model is easy! All you have to do is make a REST call to the “Train a Dataset” endpoint. Again, you can use cURL or write wrapper functions in your language of choice. You just need to provide a name for the new model and the id for the Dataset the Model will be generated from.

The service returns a JSON object with some information about how the model is being trained, the dataset its using, its current status (it can take a long time to create a model if there are a lot of pictures in your dataset), and most importantly, the model id. To see if the model is ready, send a request to the Get Training Status endpoint.  When it’s ready, you can get some information about your Model’s accuracy by sending a request to the Get Model Metrics endpoint.

Once the Model is ready, you can copy the Model id back into your Salesforce Apex logic and classify images using the new Model!

Let’s Try It Out!

To use the new model, we’ll go back into our Salesforce org and pull up the controller class for the sample Visualforce page we made. Then in the getCallVisionUrl() method, we’ll change the final line so that it passes in our new Model’s id instead of "GeneralImageClassifier" and a URL to the image we’re trying to classify. It will look something like this:

return Vision.predictUrl('http://myimageurl.jpg',access_token, 'myModelId');

Nailed It!

Nailed It!

Our new model is 99.174% sure this is a picture of an X-Wing. With this information, we can now build out a more useful page that populates a service form with potential replacement parts, reputable part vendors, cost estimates, and all sorts of other information we already have in our Salesforce org. We’ve already learned a lot about predictive vision in this post, though, so the full integration of Predictive Vision with our estimate app will have to wait for another post.

NEXT TIME: Integrating into Lightning Components

Topics: Salesforce