topic Re: How to split or cut overlong SAS program files automatically in SAS Programming

How to split or cut overlong SAS program files automatically

KidCat — Tue, 24 May 2022 06:47:23 GMT

Hello all,
Today I am posting here with a rather unusual question.
I have run into a project that uses overlong SAS programs. Lengths over 10,000 lines are not uncommon (I know there are waaayyy longer programs....). These programs usually consist of functional modules; some of these programs are pure lists of encapsulated macros without a call. My question is: How can such SAS program files (not datasets!!) be automatically split into smaller, more manageable SAS program files? Have you ever done something like this? Using tags?

Looking forward to learn from your experiences!

Re: How to split or cut overlong SAS program files automatically

andreas_lds — Tue, 24 May 2022 07:03:00 GMT

I don't know a tool that can do this and would not trust any tool that promises such miracles.

Each macro should be stored in a separate file named exactly like the macro.

Re: How to split or cut overlong SAS program files automatically

KidCat — Tue, 24 May 2022 08:39:46 GMT

I agree with you.
However, it doesn't hurt to ask if someone has done this before, ideally using SAS.

It would help to save a lot of manual work.

These "lists" contain dozends of encapsulated macros....

You know I'm thinking of an approach using tags "#" like below.

overlongmacro.sas:

#BEGIN

#NAME part1

#END

#BEGIN

#NAME part2

#END

split or cut into

part1.sas:

#BEGIN

#NAME part1

#END

and

part2.sas:

#BEGIN

#NAME part2

#END

Sounds feasible, but devil's in the details, as per ususal

Re: How to split or cut overlong SAS program files automatically

PaigeMiller — Tue, 24 May 2022 10:40:59 GMT

@KidCat wrote:

Hello all,
Today I am posting here with a rather unusual question.
I have run into a project that uses overlong SAS programs. Lengths over 10,000 lines are not uncommon (I know there are waaayyy longer programs....). These programs usually consist of functional modules; some of these programs are pure lists of encapsulated macros without a call. My question is: How can such SAS program files (not datasets!!) be automatically split into smaller, more manageable SAS program files?

Why do you want to do this? What is the harm in leaving the files un-split? What is the benefit in splitting SAS code files?

I agree with @andreas_lds , I don't think there is a way to do this other than manually, and I don't see a benefit to doing it manually either. Doing work for no benefit, most people have better things to do.

Re: How to split or cut overlong SAS program files automatically

Patrick — Tue, 24 May 2022 12:31:31 GMT

@KidCat

Because this is sort-of a "re-design" to create better organized code that's easier to read and maintain, I'd say it's worth putting in the "human" and not just trying to write some code that splits things up.

If it's a lot of macro definitions then I guess you could write some code that creates a separate file per macro ..../ and matching mend. ..but then you've got all these files so someone would actually also to create some documentation.

It's also very possible that there are multiple scripts with "copy/pasted" macro definitions so to do some analysis and streamline and document things is highly likely worth the effort and will pay-pack in the moment where you have to investigate an issue, maintain/modify code or migrate - i.e. to SAS Viya as part of platform modernization.

Such effort is of course only worth considering for code that's to stay or that creates critical business results that need to be reproducible and auditable.

Re: How to split or cut overlong SAS program files automatically

Astounding — Tue, 24 May 2022 13:39:31 GMT

Here's a list of ideas to consider, to help decide on the right path.

Within a single program, do all your macro definitions appear up front at the beginning of the program?

Your macro definitions end with a %mend statement. Do the %mend statements include the name of the macro, or just a semicolon at the end.

Do you have macro definitions nested within other macro definitions?

If program A and program B both define a macro X, is the definition of %X identical in both programs?

Depending on your answers, here are a couple of alternatives to consider:

Write a SAS program that reads one of your lengthy programs. The goal of that SAS program would be to pull out the macro definitions and write them to the like-named files, creating an autocall library.
Save the macro definitions in one huge, separate file. Then have your current programs %include that separate file instead of hard-coding macro definitions.

None of this is an ideal solution, and all require some work on your part. But they are possibilities.

Re: How to split or cut overlong SAS program files automatically

SASKiwi — Wed, 25 May 2022 20:52:13 GMT

I'm a big fan of AUTOCALL macros so one option would be to split those off into separate program files - one macro per file. If you are going to reorganise your code at all then I'd advise you to do it "properly" and comprehensively otherwise it is not really worth the effort. Splitting by common functionality is my normal approach.

Re: How to split or cut overlong SAS program files automatically

AllanBowe — Mon, 30 May 2022 00:47:32 GMT

Indeed this will be a manual job - and it's a great opportunity to consider a standardised framework for organising your SAS project(s).

The one we use at 4GL Apps is described here: https://communities.sas.com/t5/SAS-Global-Forum-Proceedings/Scaffolding-SAS-Projects-With-NPM-and-SASjs/ta-p/726347

The nice thing about this is you can have the best of both worlds - the project is modular (standalone programs, macros, jobs, services, tests, binary files) and it gets compiled into an 'overlong' SAS program, which is easy to transport / debug / reproduce.

We regularly run 250k plus programs (mainly due to special-case embedded binary files) and work quickly in the source code where programs are rarely more than a few hundred lines.

Re: How to split or cut overlong SAS program files automatically

KidCat — Mon, 30 May 2022 07:33:50 GMT

Thank you for recognising the merits of such an approach. Since we first want=need to set tags for documentation, we will also take this opportunity to set tags for "cutting". Whether this "cutting" will be done manually or automatically will be decided in the course of the project. Of course, we want to avoid having to manage two code versions (cut, uncut) for as long as possible.