topic Re: PROC OLLAMA, Running LLM Locally With SAS code in Developers

PROC OLLAMA, Running LLM Locally With SAS code

GreenCode — Wed, 09 Apr 2025 15:45:55 GMT

PROC OLLAMA, a SAS procedure written using SAS/toolkit calls llama.cpp to run the large model locally(Windows 11 x64), details: analyserxyz/myrepo1: test

Example code:

options fullstimer;
libname tmplib "c:\temp";
proc ollama out=tmplib.result;
model "c:\temp\gemma-1.1-7b-it.Q4_K_M.gguf";
params "-no-cnv -ngl 99 -p ""Once upon a time""";
run;

Output:

Re: PROC OLLAMA, Running LLM Locally With SAS code

Vrushin — Fri, 11 Apr 2025 10:46:34 GMT

is there a way to run this in viya?

Re: PROC OLLAMA, Running LLM Locally With SAS code

GreenCode — Sun, 13 Apr 2025 12:35:30 GMT

Unfortunately, there is no SAS/Toolkit module under Viya right now, so this procedure does not work on Viya. It can be replaced by other methods, such as using x, pipe to execute the llama-cli command line to output the result to a file, and then read the result back from the file.

Re: PROC OLLAMA, Running LLM Locally With SAS code

williamnadolski — Mon, 14 Apr 2025 15:23:52 GMT

This looks really promising - is there anything that can be done to help prioritize the development of this kind of development for Viya or SAS workbench?

Re: PROC OLLAMA, Running LLM Locally With SAS code

GreenCode — Tue, 15 Apr 2025 07:56:20 GMT

The more people who need this module, the more likely it is that R&D will deliver it as soon as possible, so it is recommended that you or remind all customers who need it to send a request to SAS Technical Support for SAS/Toolkit for Viya 4