This post introduces the GPT-4 Turbo with vision model from Azure OpenAI and explores its integration with SAS products like SAS Visual Analytics and SAS Studio. Imagine uploading a screenshot of a basic report and simply asking questions to understand its contents. Moreover, you can provide screenshots of your source files with sample data, and the GPT-4 Turbo with vision will generate the necessary SAS code to recreate the table in SAS Viya. Curious about turning a screenshot into a functional in-memory table in SAS Viya? Dive into the post and check out the accompanying video for a comprehensive guide.
"Wait, do you mean LLM?!?"
On December 12th, Microsoft unveiled the public preview of the GPT-4 Turbo with vision model on Azure. This Large Multimodal Model (LMM), developed by OpenAI, accepts both text and image inputs, marking an evolution from the previous Large Language Models (LLMs). Within Azure AI Studio, this model can be seamlessly integrated with Azure AI's vision services, expanding its capabilities to include:
And the list goes on.
To grasp the functionality of the GPT-4 Turbo with Vision model, it's best to walk through a series of practical examples. Consider this scenario in Azure OpenAI Studio: you upload a screenshot of a SAS Visual Analytics report.
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
From there, you can engage with the model in various ways:
Ask for identification: "What is this?"
Request data extraction: "Please extract the report data to a CSV file, comma delimited.”
Seek financial insights: "What are some key takeaways from the report?"
With LLM or LMM models you should always double check your calculations! Don’t trust the models blindly! [says Microsoft...].
Ask strategic questions: "Based on the gross profit margin – which one product would you discontinue and why?"
The response to such a question can be complex and controversial. For instance, if sofas show a low gross profit margin, it might suggest discontinuing them. However, consider the broader impact—what if sofas draw customers into the store who then purchase additional items?
My perspective is that the model's output should be viewed as a reflection in a mirror—it shows only what is directly in front of it. Similarly, the output from a model should serve as a tool to inform decision-making, not replace it.
The final decision rests on human shoulders.
Ready to take a bold step forward? Can you turn a few screenshots into a fully functional in-memory table in SAS Viya? Watch the video for a demonstration of the process.
I trust that I've demonstrated the art of the possible—converting images into a target table, it's entirely achievable.
"You are an AI assistant that helps people build data structures. Users will submit images of their target table, data set or report. You will ask them what the source data looks like. Users must respond by submitting images, describing each source table or paste a few columns with sample data. You will then generate the SAS code needed to realize the data structure. You must breakdown the task into steps, then detail what operations are required: loading files, joins, filters, sorts, summarizations, etc. At the end, the target table must be loaded in-memory, in CAS as a global promoted table."
You can script the whole approach using the OpenAI Python client library. To access the images, you can use a URL or, encode the image using base64 and pass it to the Azure OpenAI API.
Screenshots to target table? Possible with SAS Viya and Azure OpenAI GPT-4 vision. Deploy the resources, upload a few screenshots of your target report, your source files and ask the model to generate the code to create the table underlying the report.
Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about having conversations with your data. If you wish to get more information, please write me an email.
Find more articles from SAS Global Enablement and Learning here.
Are you ready for the spotlight? We're accepting content ideas for SAS Innovate 2025 to be held May 6-9 in Orlando, FL. The call is open until September 25. Read more here about why you should contribute and what is in it for you!
Data Literacy is for all, even absolute beginners. Jump on board with this free e-learning and boost your career prospects.