LLM to review SAS coding? Which is best?

[Reeza]

What is the prompt used?

[illmatic]

"I am attempting to... my source dataset is called .... here is the proc contents of said dataset: "pasted proc contents". And then of course it's being as detailed as possible in my explanation and reminding ChatGPT to ask me clarifying questions instead of assuming. Errors of course are copied from the log and pasted back for debugging, additional context is added when I notice that ChatGPT is falling into a logic trap, but that's much rarer now with 4o.

[Reeza]

Nice, prompts definitely have a big impact on the results in the LLM results. Vague questions get vague responses is very true in LLM. If you can't articulate your problem well, LLMs won't be super effective.

I'm looking at some code specific models (Bamba2) in the near future. Curious to see how those perform.

[NKormanik]

Similar to talking with a human:  If you use grade school talk, you don't get far.  If you use post grad talk, you get lots further. 

ChatGPT and Claude are able to carry on quite well at the post grad level.  Not true of most humans.

[illmatic]

I have had excellent success with ChatGPT 4o when comparing it to Claude and Gemini. It is by far the best for SAS code creation, debugging, etc.

I have also tried the plug-in for VS Code for SAS so that I may use GitHub's CoPilot, and it's also pretty good, but there were some features that were missing in the plug-in that I needed for my tasks, so I have to abandon it for now.

For your particular scenario, ChatGPT 4o can debug and work it's way out of logic issues, but the only annoying things right now for me is feeding ChatGPT background information (i.e. characteristics of a dataset, in which I will copy-paste the proc contents into chatgpt) and the lack of an AutoGPT so it can debug and test the code itself without me having to go back-and-forth. If ChatGPT could get better insight into the data structures referenced in my code + AutoGPT, I'd be in heaven.

[NKormanik]

Sometimes ChatGPT just simply falls short when it comes to advanced macros.  Terribly frustrating.  Error after error after error.

I let Claude have a look, and Claude sometimes magically finds the solution.  Unreal.  No errors in the execution.

But, generally ChatGPT seems to have an edge over Claude.  And!!!  ChatGPT allows attaching more types of stuff.  INCLUDING SAS DATASETS THEMSELVES!!!!