Intro
The Assistants API is a new feature in Azure OpenAI. It’s currently (at the time of writing – April 2024) in preview and the documentation is, shall we say, still in development.
But being in preview with questionable documentation isn’t enough of of a reason not to use this tool, because it really is very good, and we can quite easily build very handy chat bots within Power Apps.
So a few weeks ago I set about building a custom connector for the Assistants API. It doesn’t have every single endpoint – some of them which aren’t needed to build a functional chat bot aren’t included, but I got the basics in here and I also put a couple of endpoints from the Files API to upload and retrieve file content for Retrieval Augmented Generation (RAG)
I’ve had to stop development of this because I no longer have access to Azure OpenAI (😭) but I’ll continue if I regain that.
In this article I’m going to assume you already created a deployment in Azure Open AI using a model that’s compatible with the Assistants API. To get started, check out this guide: https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/assistant
Installing the Custom Connector
Here’s a link to the custom connector swagger and code in my GitHub: https://github.com/wilheimpage/AzureOpenAIAssistantsCustomConnector
To use this custom connector yourself, download the swagger JSON from GitHub and change the URL in the “host” property from YOUR_RESOURCE_NAME.openai.azure.com to whatever you called your resource when you created it:
In Custom Connectors, which is accessible via the Power Apps or Power Automate portals, click New custom connector then Import and OpenAPI file:
After uploading and saving the custom connector, click 5. Code in the breadcrumbs, enable code using the toggle and paste or upload the contents of code.cs from the GitHub repo. In the Operations drop down, select GetAssistants, GetFileContent and GetMessages
I’ll explain why this code exists..
Any Assistants endpoint that returns an array of objects, like Get Messages or Get Assistants, has an array property of the response called “data”. I found that Power Apps doesn’t like this. If you upload sample data for the response that contains this property and then test, it works fine in the custom connector setup and the schema validation passes, but in Power Apps, intellisense won’t auto-complete any of the properties of the response and when you run it, you get some weird schema error.
There are two workarounds for this. One of them is to simply not define the response in the custom connector setup. Then you end up with an untyped object in Power Apps you need to parse manually in Power Apps. This is more than a little fiddly and annoying so I went with option 2 – rename the data property using code in the custom connector setup. In the case of Get Messages and Get Assistants the “data” property is renamed “messages” and “assistants” respectively. Using the new schema in an endpoint definition, it works perfectly in Power Apps. Here is a screen shot of the code showing how the data property is renamed to “assistants” for the Get Assistants endpoint.
The 3rd endpoint being processed by the custom code is Get File Content. The reason for this is because we need the response from the API to be understandable by the Image control in Power Apps. By default the Azure Open AI Files API returns the binary content but the Image control needs a data URI. You could do this in a flow, but in my application I want to display the images in the same gallery as the messages. The Image control’s .Image property isn’t a behaviour property so it can’t trigger a flow. Therefore it needs to be able to get its content from the custom connector directly.
So, to do this we need to turn the stream response from the API into a base64 string:
This gives us a base64 string in the body of the response. Not a JSON blob object or anything like that, just a plain old string of base64. Within the app, you need to construct a data URI. The content type is missing from the response, so you need to infer that somehow, or set it statically if you like to live dangerously:
The code above is from the .Image property of an Image control that’s inside a galley which has the Get Messages endpoint as its Items. Each object within the messages array has a content property which is itself an array, containing the text response from the AI and also, if you’ve asked it for one, an image. This could be a chart or graph you’ve asked it to construct from the data you uploaded.
The corresponding image file is referenced by its ID which you can then use to hit the Files API to get the content. I’ve just hard coded image/jpeg as the content type, which works for charts and graphs the AI makes, but if you want to ask it for another kind of file you may need to work on that bit yourself (e.g using the PDF control to display PDFs you want it to generate etc.).
How to use it
Create a canvas Power App and add the custom connector as the data source.
If you created an assistant in the Open AI Assistants Playground in the Azure Open AI studio, then you should be able to retrieve its details using the GetAssistants action like this to return an array you can have in a gallery, drop down, combo box or list box etc:
OpenAIAssistants.GetAssistants().assistants
If not, then you can create one using input boxes to fill the required parameters of OpenAIAssistants.CreateAssistant:
Use UpdateAssistant() to allow users to edit the system prompt via a text input box:
You need to create a thread before you can add a message to it. The thread is the conversation between you and the assistant. So the process goes like this:
Create thread -> add message to thread -> create a run -> wait for run to finish -> get thread -> add message to thread -> create run -> get thread -> etc etc
So the action of creating a message needs to be aware of whether a current thread exists or it needs to invoke a new one. Here’s some code in the OnSelect of a “send” icon alongside the text input for a new message.
txtMyChatMessage here is the input box. As you can see, a timer is started at the end of this process. Here’s its OnTimerEnd property:
Each run of the timer gets the contents of the thread and puts it in a variable called aChat, gets the status of the run and then if the status is something terminal, turns the boolean variable that starts the timer back to false.
The timer will repeat as long as the status is non-terminal. Here’s its Repeat property:
Now you can use the variable aChat as the Items property of a variable height galley with a text label inside set to Autoheight = true with the Text property being:
Position the text label in the gallery, justify the text and colour the background based on the role property of the item to make a chat-bot looking conversation window, with your messages on the right and the bot’s on the left.
If you want to add files to an assistant for RAG, you can use the CreateAssistantFile action for this:
In this example I used the Add picture control. It actually supports all kinds of files so you can use it for uploading Word docs, CSVs and any other file type that’s supported by the assistants API. It’s a two-step process, firstly upload the file to the Files API then using the id of the newly uploaded file, attach it to the assistant. I made a flow to handle the upload to the Files API, so in this formula we invoke the flow in the second input of the function. The function will wait for the flow to return the ID of the file before attaching the file to the assistant. The flow looks like this:
See this article for more information about this flow: https://willpage.dev/2019/12/20/using-the-http-action-to-post-multipart-form-data-in-power-automate-logic-apps/
Finally
I used this app to upload a couple of files – a project scope of works plus a CSV containing all the time entries on the project. You can ask the assistant to analyse the two and figure out why overruns have occurred on certain tasks or milestones or what the cause of any rework was. It’s a lot faster than reading through hundreds of time entries yourself and trying to figure it out.
Anyway, that’s just about it. Questions welcome in the comments. I won’t be able to offer too much help on this because I no longer have access to the API!
If you like my content and want to support the work that I do, you could leave a small donation. Watermelons get quite expensive out of season in New Zealand.
William's IT Blog 🍉 