topic Re: Approach to long and complex SAS coding in SAS Programming

Approach to long and complex SAS coding

ammarhm — Sat, 16 Sep 2017 03:55:02 GMT

I have a question regarding your approach to complex project / long SAS code files.

One problem that I frequently encountre with long SAS projects, is that the code becomes so long that it becomes difficult to come back at a later stage and just jumt into a specific pasrt and debug it or change it...

Commenting the code is something that I always use, but still, this becomes impractical when the code is very long...

I wish that SAS could add the option of Bookmarks into the code so that it would easier to find the part I want to go to and debug from there.

HOWEVER, my approach now is to split the code, so I have one file that prepares the main dataset, and then several other .sas files with code that basically built from the main dataset to give a specific part of the solution.

This is not a straight forward solution as you can imaging, my question to you is: Is there a code/way to excute the code in a .sas file from another .sas file

What I am trying to do is something somilar to %include but that would excute standard sas code rather than a Macro code.

So here is what I want to do

some code to excuse data.sas which prepares the main dataset
.
.
.
.
code here to process the data generates
..
.
.

I am also interested to hear what you do or if you have a different approach to long/ complex sas code files

Thnak you

Re: Approach to long and complex SAS coding

Reeza — Sat, 16 Sep 2017 04:02:38 GMT

%INCLUDE() runs whatever code is included, whether it's SAS code, formats, functions or macros.

I typically use the approach there but my code is organized into sections based on project.

Formats & Functions each go into their own file.

Macros - it depends, depending on the type will either be their own file or multiple in one. If it's very project specific, they'll be in one, if they're more generic they're in a single unique file.

This approach allows me to re-use macros/formats/functions easily.

I then break my main programs into:

Data Import

Data Prep/Cleaning

Data Analysis

Reporting & Graphs

Re: Approach to long and complex SAS coding

Reeza — Sat, 16 Sep 2017 04:02:57 GMT

Oh...an there's a main 'control program' that drives the whole process.

Re: Approach to long and complex SAS coding

AlanC — Sat, 16 Sep 2017 04:37:56 GMT

I concur with what Reeza said about breaking it up with some of my own caveats.

%include works for anything: don't think of it as macro coding because it isn't (despite the % in front).

For complex coding, I prefer to use %include and data step vs macro. Why? Easier to debug. An example:

data _null_ ;

file someTempFile ;

put 'data test;'

/ 'set sashelp.class;'

etc.

run;

%include someTempFile ;

What you get is standalone sas programs that can be tested in isolation vs macro stuff. When macro gets really complex, it is very hard to debug. Using the put staments, you achieve the same processing cost but at leat you can see what is happening.

Split it, isolate it, make it something you can easily decipher and test.

Also, adopt documentation standards. See an example on sasCommunity called SaviDoc. File link is gone (I can send you it) but the standards are listed. If you follow them, you can create your own bookmarks by merely parsing your SAS code.

Re: Approach to long and complex SAS coding

Patrick — Sat, 16 Sep 2017 07:01:08 GMT

@ammarhm

I normally try to break up the problem into logical units and to implement these as self contained SAS programs.

This allow to test each program individually as well as to run the whole project either via a master.sas or even better via a scheduler which then also allows for parallel run of programs with full control over job dependencies.

Re: Approach to long and complex SAS coding

ammarhm — Sat, 16 Sep 2017 07:23:56 GMT

Thank you everyone for the responses which are highly appreciated.

@Patrick could you please elaborate on using a "scheduler" ? I am interested in knowing how you use this
Kind regards

Re: Approach to long and complex SAS coding

SASKiwi — Sat, 16 Sep 2017 08:11:24 GMT

Here is a useful link that describes the various ways SAS jobs can be scheduled. It is a very large topic so don't be put off by this. Scheduling is an essential part of any SAS Production environment.

http://support.sas.com/documentation/cdl/en/scheduleug/68697/HTML/default/viewer.htm#n1av1t0k29q1rfn1r9cl5bg4dftb.htm

Also for managing large and complex SAS applications, using version control tools is also very helpful as you can compare changes made between program versions and easily follow the complete history of your code development.

Re: Approach to long and complex SAS coding

ammarhm — Sat, 16 Sep 2017 08:29:34 GMT

@SASKiwi

Thank you for the reply

What version control tools do you use / recommend please?

Kind regards

Re: Approach to long and complex SAS coding

AlanC — Sat, 16 Sep 2017 12:50:22 GMT

Use Git. I wouldn't even bother with others (or even looking at alternatives) unless you are required to use something else.

Git is also integrated with EG so that helps a lot as well.

Re: Approach to long and complex SAS coding

MichelleHomes — Sat, 16 Sep 2017 14:06:52 GMT

You might find this thread on Git helpful https://communities.sas.com/t5/SASware-Ballot-Ideas/enable-non-local-git-server-integration/idi-p/312736

Re: Approach to long and complex SAS coding

Quentin — Sat, 16 Sep 2017 16:20:20 GMT

Modular programming is definitely a big help to keeping things organized. You can use %INCLUDE statement for modules (as others have pointed out, it's not related at all to the macro langauge), or macros. I tend to use macros for modularization, because the definition I learned for a macro was "a parameterized unit of code." That ability to pass parameters makes it easier to develop, test, and re-use.

Ed Healton has a nice paper on an approach to top-down structured programming using macros:

http://www.lexjansen.com/nesug/nesug01/at/at1010.pdf

When you do it well, you can end up with a nice high-level outline of macro calls, which make it easier to understand the structure / flow of the of the code wiout stating at pages of code. Ed mentioned to me at one point that his rule was "if you can't read the whole module in a single screen, consider splitting it into multiple modules."

So you might end up with:

%setup()

%ReadData(file=)

%AnalyzeData(...)

%EmailResults(..)

%CheckLog(...)

And of course each module above would have sub-modules.

In Enterprise Guide, you can use the process flow view to visually arrange the modules (where each module is a linked .sas file storing a single macro definition).

I don't always use this approach to the extreme, but it's handy to keep in mind for complex problems. And when I use it, I rarely regret it.

--Q.

Re: Approach to long and complex SAS coding

AlanC — Sat, 16 Sep 2017 19:05:59 GMT

I use macros the same but I try not to venture too deep behind simple call
and parm substitution.

Re: Approach to long and complex SAS coding

SASKiwi — Sat, 16 Sep 2017 22:40:16 GMT

@ammarhm - Regarding version control tools - I always recommend checking out those that are already used in your organisation. Perhaps there is already a preferred enterprise tool that is already setup and all you have to do is start using it. That is the case where I currently work. It just so happens that the tool is Microsoft Visual Studio with MS Team Foundation Server. It works really well with SAS programs and integrates nicely with SAS Enterprise Guide.

I am sure there are many other VC tools that work equally as well but using an existing tool in your company is just so much easier.

Re: Approach to long and complex SAS coding

ammarhm — Sun, 17 Sep 2017 01:41:40 GMT

Thank you everyone for the suggestions and useful answeres

Re: Approach to long and complex SAS coding

bentleyj1 — Mon, 09 Oct 2017 20:02:17 GMT

We have to use Subversion for version control... very much overkill for what we need to do. I use UltraEdit. It automatically makes a backup copy identified by a version number each time you save.

Re: Approach to long and complex SAS coding

don_mcmahon — Fri, 13 Oct 2017 17:20:50 GMT

I too use the approach of organizing my programs by import -> coding -> reports -> stats -> graphs.

To organize, each section and subsection starts with a comment with two asterisks in outline format. Example:

** 1.0 Import source data;

** 1.1 Import Questionnaires;

** 1.2 Import Labs;

...

** 2.0 Recode and transform data;

** 2.1 Collapse categories;

** 2.2 Normalize skewed distributions;

I then have a separate program that takes the name of a SAS program as it's input and then prints out the program outline as defined by the text following the double-asterisks. This is simple text parsing, but produces a document that provides a road map to the entire program. It doesn't help with the problem of running specific sections of code. But helps a when you need to understand a program you wrote a long time ago.

...don

%macro pgmdoc(fn);
filename pgmfile "&fn";

data _pgmdoctemp_;
attrib Outline length=$255 format=$145.;
infile pgmfile linesize=255;
input Outline &;

if scan(upcase(Outline),1) ='DATA' then do;           Outline='        '||trim(Outline); output; end; else
if scan(upcase(Outline),1) ='PROC' then do;           Outline='        '||trim(Outline); output; end; else
if scan(upcase(Outline),1) ='SET' then do;           Outline='            '||trim(Outline); output; end; else
if scan(upcase(Outline),1) ='MERGE' then do;       Outline='            '||trim(Outline); output; end; else
if scan(upcase(Outline),1) ='LIBNAME' then do;        Outline='        '||trim(Outline); output; end; else
if substr(upcase(Outline),1,4)='%LET' then do;        Outline='        '||substr(Outline,1,length(outline)); output; end; else
if substr(upcase(Outline),1,8)='%INCLUDE' then do; Outline='        '||substr(Outline,1,length(outline)); output; end; else
if substr(Outline,1,3)='** ' then do;               Outline=trim(substr(Outline,4,length(Outline)));output; end;
run;

title 'SAS Program Documenter';
title2 "&fn";
proc print data=_pgmdoctemp_ noobs;
var Outline;
run;
%mend;

Re: Approach to long and complex SAS coding

Reeza — Fri, 13 Oct 2017 19:16:30 GMT

@don_mcmahon You may want to take a look at PROC SCAPROC if you aren't already familiar with it.

Re: Approach to long and complex SAS coding

Peter_L — Wed, 22 Nov 2017 13:40:01 GMT

I use most of the methods mentioned by other contributors plus a few others.

Break code into cohesive sections.

Make the sections have as low a coupling as possible.

Use macro variables to parameterise code. There is a downside: too many global variables (= coupling).

Save code as "modules" and %include them as needed.

Write pre- and post- assertions for each module formally to define what it does.

Include the unit test code with every module commented out at the end.

I write the documentation in the code marked with special comment symbols of /** or **; A simple awk script then processes these into html in a similar way to javadoc.

Replace as much DATA step merge code with PROC SQL as possible. It makes the code shorter and easier to read.

Use data-driven methods as much as possible. This tends to make things simpler, shorter, more reliable and easier to test. SQL helps here.

Replace macro logic with DATA step logic and data-driven methods. DATA step logic is far easier to debug and quoting is not a problem.

To make your code easier to read change the editor to show a light grey background to comments. The colour extends to the screen width. Then use comments to break up sections of code. You will be surprised how big a difference it makes.