%include thwarts data flow diagram from SCAPROC

[MartyS]

Hello,

I am relatively new SAS convert tasked with taking ownership of some legacy production code.  I recently wrote some SAS code to ingest the results of SCAPROC and produce some nice diagrams.  I started with the ds2const macro, but the user interface was rather difficult, so I wrote a replacement that feeds the graphviz suite.  While I am documenting the code, I need to be as minimally-invasive as possible.

Everything worked fine, except it appears that a file called in via %include will be documented as a unit, instead of the individual datasteps.  Let me explain.  It turns out the production code I am looking at has wrapper code which calls %include for the real code.  We added PROC SCAPROC to the real code and the resulting file had 1049 "/* JOBSPLIT:" lines consisting of various CATALOG, SYMBOL, LIBNAME, FILE and DATASET entries, followed by the final lines:

/* JOBSPLIT: ELAPSED 20307685  */
/* JOBSPLIT: PROCNAME DATASTEP */
/* JOBSPLIT: STEP SOURCE FOLLOWS */

/* JOBSPLIT: END */

After some testing, it appears that SCAPROC will refuse to log any SAS file that has been included via %include.  I could always make changes to the production code and the way it is invoked, but I am trying to avoid that while I document it.

Is there an easy was to tell SAS to analyze the individual steps within included files?

Many thanks,

Marty

[ArtC]

Have you turned on the SOURCE2 system option?  This option allows the display of included source code in the LOG.

[MartyS]

Thanks for the fast reply.  I just now tested SOURCE2 and it did as expected, putting the included source into the log.  Sadly, SCAPROC still treats the included file as a single unit of work.

Thanks again.

[ArtC]

Then I do not understand the issue.  The SOURCE2 option should surface code that is included from within a macro.  What do you mean that the SCAPROC treats the included file as a single unit of work?  Is it not shown in the LOG? Are you not seeing the execution notes?

[MartyS]

The source is in the log, but not the output of SCAPROC.  SCAPROC lists only a single "step" in its output, even though the log shows multiple steps.  Does that make sense?

Thanks,

Marty

[ArtC]

The results of most procedures are not written to the LOG, but are written to an output destination, such as the OUTPUT window.  Are you saying that the your program is not working or that the SCAPROC procedure is not showing all of the steps in your program?

[MartyS]

Thanks for working with me on this.  Yes, SCAPROC is not showing all of the steps, but only when called via %include.

For example, the following PROCNAME entries are in the SCAPROC output when the code is called directly:

vpds0132a[~/prj/sas_dfd]$ grep JOBSPLIT:.PROCNAME scagraph_dot.txt        

/* JOBSPLIT: PROCNAME DATASTEP */

/* JOBSPLIT: PROCNAME PRINT */

/* JOBSPLIT: PROCNAME SQL */

/* JOBSPLIT: PROCNAME PRINT */

/* JOBSPLIT: PROCNAME SQL */

/* JOBSPLIT: PROCNAME PRINT */

/* JOBSPLIT: PROCNAME SQL */

/* JOBSPLIT: PROCNAME PRINT */

/* JOBSPLIT: PROCNAME SQL */

/* JOBSPLIT: PROCNAME PRINT */

/* JOBSPLIT: PROCNAME SQL */

/* JOBSPLIT: PROCNAME PRINT */

/* JOBSPLIT: PROCNAME SQL */

/* JOBSPLIT: PROCNAME PRINT */

/* JOBSPLIT: PROCNAME SQL */

/* JOBSPLIT: PROCNAME PRINT */

/* JOBSPLIT: PROCNAME SQL */

/* JOBSPLIT: PROCNAME PRINT */

/* JOBSPLIT: PROCNAME SQL */

/* JOBSPLIT: PROCNAME PRINT */

When the code is called via another script that only includes this code, we see the following results from SCAPROC:

vpds0132a[~/prj/sas_dfd]$ grep JOBSPLIT:.PROCNAME scagraph_dot.wrapper.txt

/* JOBSPLIT: PROCNAME DATASTEP */

Does that help?

Thanks again,

Marty

[ArtC]

Thank you for your patience, I believe that I now understand.  While I have only a passing understanding of this procedure, I believe that neither macro language elements or (apparently) iincluded code is passed to the Code Analyzer.  Perhaps someone who has a better understanding of this procedure will join the conversation.  I did not find any options that changed this behavior.

Perhaps there is an alternate approach to the problem.  What specifically are you trying to get from the procedure?  A list of steps?

[ArtC]

I experimented and here is a work around.  There is probably/must be a better way.

1.  surround the code that has the %INCLUDE with a dummy macro This code will also have your PROC SCAPROC steps.

%macro doit;

..... some code ....

%include yourcode;

. . . . . other code . . . .

%mend doit;

2. Turn on the MFILE system option and create a MPRINT fileref

filename mprint 'somefile.sas';

options mprint mfile;

3. execute the macro

%doit

4. reexecute the file generated by the MFILE option (SOMEFILE.SAS).  This code will have no macro language references and the %INCLUDEs will have been executed.  This means that the SCAPROC will do what you want.

[MartyS]

ArtC, thanks for all of your help.

>  I believe that neither macro language elements or (apparently) iincluded code is passed to the Code Analyzer.

I am seeing some strange stuff with macros as well as %include.  I just noticed the macro issues yesterday as I was testing something else.

ArtC wrote:
I experimented and here is a work around.  There is probably/must be a better way.
1.  surround the code that has the %INCLUDE with a dummy macro This code will also have your PROC SCAPROC steps.
%macro doit;
..... some code ....
%include yourcode;
. . . . . other code . . . .
%mend doit;
2. Turn on the MFILE system option and create a MPRINT fileref
filename mprint 'somefile.sas';
options mprint mfile;
3. execute the macro
%doit
4. reexecute the file generated by the MFILE option (SOMEFILE.SAS).  This code will have no macro language references and the %INCLUDEs will have been executed.  This means that the SCAPROC will do what you want.

I may have to head down this path.  The challenge is that the code is production, with several side effects including making changes to production datasets, triggering other code to run and sending email messages, so I need to be very careful about running it outside of production.  Ironically, the reason I am doing this is to document and understand what this code does.  In other words, to document what this code does, I need to introduce risk into production.

> What specifically are you trying to get from the procedure?

I tried to attach a PDF or SVG of an example flow diagram I made of a non-production SAS job, but it looks like the forum won't let me.  I want to make a diagram of what this code ingests, what intermediate datasets are produced, and what are the outputs.

At this point I think that I might be trying too hard to make this work.  I probably would be better served with a pot of coffee and an afternoon with Visio.

Thanks again,

Marty

[MartyS]

ArtC,

My apologies, I was looking in the wrong place.  Attached is a sample PDF produced from some simple code that shows what is going in and what is going out.  This is a dataflow diagram of the code that reads the SCAPROC output, making an output file for input into graphviz.  I had to unroll a macro in order to get SCAPROC to work correctly.

> What specifically are you trying to get from the procedure?  A list of steps?

I want a diagram, similar to the attached one, for a particular SAS job.  The diagram will be used for me and for explaining the code to others.  Perhaps just as important, I want to generate these diagrams programmatically for our other 300-ish legacy jobs.

Thanks again,

Marty

[PhilMason]

I just found this post, and invented a similar system for making data flow diagrams. I'd be interested to know if you found a solution to your problem of how the %include seemed to be excluded from Proc SCAPROC. You should try to contact Eric Thies (I thinkt that's his name) as he is the guy who made this PROC and could best comment on it. - Phil