LLM to review SAS coding? Which is best?

[Reeza]

What is the prompt used?

[illmatic]

"I am attempting to... my source dataset is called .... here is the proc contents of said dataset: "pasted proc contents". And then of course it's being as detailed as possible in my explanation and reminding ChatGPT to ask me clarifying questions instead of assuming. Errors of course are copied from the log and pasted back for debugging, additional context is added when I notice that ChatGPT is falling into a logic trap, but that's much rarer now with 4o.

[Reeza]

Nice, prompts definitely have a big impact on the results in the LLM results. Vague questions get vague responses is very true in LLM. If you can't articulate your problem well, LLMs won't be super effective.

I'm looking at some code specific models (Bamba2) in the near future. Curious to see how those perform.

[NKormanik]

Similar to talking with a human:  If you use grade school talk, you don't get far.  If you use post grad talk, you get lots further. 

ChatGPT and Claude are able to carry on quite well at the post grad level.  Not true of most humans.

[illmatic]

I have had excellent success with ChatGPT 4o when comparing it to Claude and Gemini. It is by far the best for SAS code creation, debugging, etc.

I have also tried the plug-in for VS Code for SAS so that I may use GitHub's CoPilot, and it's also pretty good, but there were some features that were missing in the plug-in that I needed for my tasks, so I have to abandon it for now.

For your particular scenario, ChatGPT 4o can debug and work it's way out of logic issues, but the only annoying things right now for me is feeding ChatGPT background information (i.e. characteristics of a dataset, in which I will copy-paste the proc contents into chatgpt) and the lack of an AutoGPT so it can debug and test the code itself without me having to go back-and-forth. If ChatGPT could get better insight into the data structures referenced in my code + AutoGPT, I'd be in heaven.

[NKormanik]

Sometimes ChatGPT just simply falls short when it comes to advanced macros.  Terribly frustrating.  Error after error after error.

I let Claude have a look, and Claude sometimes magically finds the solution.  Unreal.  No errors in the execution.

But, generally ChatGPT seems to have an edge over Claude.  And!!!  ChatGPT allows attaching more types of stuff.  INCLUDING SAS DATASETS THEMSELVES!!!!

[ajscott20090]

As of this writing, Google Gemini has been far better than ChatGPT. I've posed very complex requests and Gemini gets it right nearly every time. Usually, there is one command that is outdated or doesn't work in that instance. With that correction, I have been very pleased with Gemini, especially compared to ChatGPT, which seems to keep guessing at answers and does not have solid logic when there are errors thrown. Gemini and Claude pulled far ahead ChatGPT after this summer, which was the topic of a news story I read just before switching to Gemini, and for coding, it is my go-to AI that I am willing to pay for monthly. 

I think each AI will pull ahead at times as they all continually improve. At the present time, the winner by far is Gemini. I haven't tried Claude again in the past month, so it may have pulled ahead. For now, Gemini seems to be the most intuitive tool, understanding what I am asking and providing the correct response on the first try (with one or two minor tweaks), after I have found my errors in my own dataset and fixed them of course!

I hope that is helpful.