Autocall Macro versioning

[KevinViel]

Does anyone have a reference or citation for a Version Control System for Autocall macros?  When we update an Autocall macro, the version number is not part of the filename or macro name (required to match to be a SAS Autocall macro).  In this way, our programs that call these macros remain uniform and require minimal, if any, updates, between projects or deliveries.  It would make sense to report the version number to the log much as we report the path and name with the SAS System Option MAUTOLOCDISPLAY.  The problem is that if we need to replicate the results, then we have to call the correct version of the macro.  We could read the programs and logs then substitute the versioned macro name.

Any ideas or cautionary tales are appreciated.

Thank you,

Kevin

[ballardw]

If you want something to appear in the Log then easiest might be a %put statement in the macro with the needed information:

%macro dummy();

%put Macro: Dummy Version=XXX  ;

<remainder of macro code>

%mend;

Though you may get tired of seeing that if you have a macro that is used very frequently. Which might make an optional parameter to show the version information useful and only execute the %put when that parameter is set.

[Quentin]

Hi Kevin.  If you don't want the log message every time you call the macro, you can add a %PUT statement to note the version number before the %macro statement.  So it can look like:

%put NOTE: Compiling %nrstr(%%)dummy (v 1.0) from autocall library ;

%macro dummy();

  %* <remainder of macro code> ;

%mend;

The first time the macro %dummy is called in a session, SAS essentially %includes dummy.sas, so the %PUT statement will execute.  

Sometimes I will add the path to the .sas file to the note as well.

I think something like that helps to get the version number in the log.  Then using git or whatever version control on the actual .sas files.

[KevinViel]

Quentin,

  Good suggestions.

  I had been including a SYMEXIST check to run a %DO-%END block that creates the global macro variable of interest and the one-time code if the macro variable did not exist.  That would give the programmer, for whatever reason, the chance to SYMDEL that macro variable to display the "one-time code" again.  I updated the AUTOEXEC.SAS to include the SAS System Option MAUTOLOCDISPLAY.

  This might actually resolve because I understand that the FDA no longer wants (accepts) externally defined macros in the submitted programs, so we may have to execute the program, use MFILE (or some run of the mill MAC_DEBUG_MFILE macro freely available in the PharmaSUG archives of papers 🙂 ), then Validate that "program" with the generated SAS code and no macro calls.  That will turn on the SOP/WI of the company.

That is not a visually appealing approach, because MFILE "spits" out the SAS code devoid of all of my indentations and formatting (screens and fonts are getting bigger and I still cannot see "1" versus "l" in code, like names 🙂  I read formatted programs "faster".

BTW, that means I omit some macro code in my setup.sas files.  Instead of using %LET statements, I define a macro, include a data step with CALL SYMPUTX(), with the third argument "G", and call the macro at as the last executed code of the program.  That way I might capture the code for a "stand-alone" program.  This is all a (stalled) work-in-progress.

%macro test ( ) ;
  %let test = one ;

  data _null_ ;
    call symputx( "test2"
                , "one"
                ) ;
  run ;

%mend  test ;

%mac_debug_mfile
  ( mcall = test()) ;

22   %mac_debug_mfile
MAUTOLOCDISPLAY(MAC_DEBUG_MFILE):  This macro was compiled from the autocall file G:\SAS Data\02_macros\mac_debug_mfile.sas
23     ( mcall = test()) ;
MPRINT(MAC_DEBUG_MFILE):   ;
MPRINT(TEST):   data _null_ ;
NOTE: The macro generated output from MPRINT will also be written to external file C:\Users\KEVIN~1.VIE\AppData\Local\Temp\66\SAS Temporary Files\_TD13100_VC-SASPROD_\#LN00010 while OPTIONS MPRINT and MFILE are set.
MPRINT(TEST):   call symputx( "test2" , "one" ) ;
MPRINT(TEST):   run ;

NOTE: DATA statement used (Total process time):
      real time           0.00 seconds
      cpu time            0.01 seconds


MPRINT(MAC_DEBUG_MFILE):   options nomprint nomfile

NOTE: The file/infile MPRINT is:
      Filename=C:\Users\KEVIN~1.VIE\AppData\Local\Temp\66\SAS Temporary Files\_TD13100_VC-SASPROD_\#LN00010,
      RECFM=V,LRECL=32767,File Size (bytes)=83,
      Last Modified=08Nov2024:09:43:07,
      Create Time=08Nov2024:09:43:07

NOTE: 4 records were read from the infile MPRINT.
      The minimum record length was 5.
      The maximum record length was 33.
NOTE: 4 records were written to the file MPRINT.
      The minimum record length was 5.
      The maximum record length was 33.
NOTE: DATA statement used (Total process time):
      real time           0.00 seconds
      cpu time            0.00 seconds



NOTE: The file TYPE is:
      Unnamed Pipe Access Device,
      PROCESS=type "C:\Users\KEVIN~1.VIE\AppData\Local\Temp\66\SAS Temporary Files\_TD13100_VC-SASPROD_\#LN00010" | clip,
      RECFM=V,LRECL=32767

NOTE: 0 records were written to the file TYPE.
NOTE: DATA statement used (Total process time):
      real time           0.21 seconds
      cpu time            0.00 seconds


NOTE: Fileref TYPE has been deassigned.
NOTE: Fileref MPRINT has been deassigned.

/* On a Windows OS, after running the code in interactive mode, typing Cntl-V produces */
data _null_ ;
call symputx( "test2" , "one" ) ;
run ;

Kind regards,

Kevin

[Tom]

This comment is worth discussion:

This might actually resolve because I understand that the FDA no longer wants (accepts) externally defined macros in the submitted programs, 

You can use other methods in SAS to generate code, not just macros.  For example you could use a data step to write the desired code to a file and then %INCLUDE the file.  This can also be done outside of SAS to generate the code.

This will also be a problem for FDA submission done using other programming languages.  Are you not allowed to define functions in R? Packages?

[yabwon]

If you have it as a SAS Package, the following snippet:

proc printto log = "/location/for/code/HereGoesMyCode.sas";
run;

%previewPackage(PackageName,*)

proc printto log = log;
run;

creates an "FDA ready" file that contains all code in the package, information about version, timestamp and authors. And it needs only 2 trivial edits: commenting out the first ~10 lines, and commenting out the last ~5 lines 🙂 And keeps files formatting 😉

Bart

[AhmedAl_Attar]

Hi @KevinViel 

For repeatability, How about adopting the following?

Version Control: Git/GitHub/GitLab/Bitbucket (pick your personal/corporate choice)

SAS Macros: Use Stored compiled SAS macros https://www.lexjansen.com/pharmasug/2006/ApplicationsDevelopment/AD05.pdf

Check-in your SAS compiled Macros Catalog (Binary File) into your Version Control Repository and tag it with a Release ##   

This will allow and your team to Check-out/Clone the specific Release(s) as and when needed  

Hope this helps

[Tom]

Your version control software should handle that for you.

You just need to know when the program ran.  Then use that information to get the version that was current at that time.

[Quentin]

I don't think it's quite as simple as saying "use version control." Would be interested to hear more about how people implement this.

So take the scenario where a company has a shared library of corporate autocall macros, and then there are multiple projects being worked on by different developers.

I'm curious @AhmedAl_Attar  and @Tom , if you can say more about the setup you would propose.

Typically a shared macro library would be stored in a location accessible to all programmers. So in some directory you have /CorpMacros, which could have .sas files (or compiled macros I guess), and then users would point their code to that directory (or it would be defined in a shared config file or whatever).  I think in this scenario, individual programmers would never need to check out macro code from a repo.  An admin would be responsible for maintaining the code in /CorpMacros.  In that setting, it's not a trivial step to roll back a macro to a prior version, because doing so would roll back the macro for every programmer in the company.

I suppose a different approach would be to keep your CorpMacros in git or whatever version control. And each project could have a local (the project) directory for storing its own collection of the CorpMacros.  Meaning for each project, the programmer(s) for that project would be responsible for deciding when they want download macros from the version control storage.  With that setup, each project could decide when to download an updated version of the a corporate macro.  So each project would basically have its own independent copy of the corp macro library.  Project A could be running %FOO v1, while Project B is running FOO v2.

I guess with the second approach, it's more work for the programmers working on the projects.  But it does give them control as to which version of a corp macro is being used for their project. And makes it easier for Project B to revert to an older version of %FOO any time they want.  In a sense this approach doesn't really have a corp macro library which is accessible to SAS.  Instead the corp macro library is stored only in version control.  And each project is responsible for maintaining a project macro library by pulling files from repository.  One down side of approach #2 is that if a developer makes a critical bug fix to  %foo, they then need to tell every project to make sure they pull down the latest version of the macro.

While I'm  fan of corporate libraries, and I was responsible for maintaining one for years at a prior employer,  I think it's a tricky task to administer properly.  And as a user of corporate macros, it always felt a bit weird to know that a macro I call on  Tuesday might do something different than it did on Monday, if the corp macro czar had updated it for some reason.  For some critical projects, I would take a snapshot of the corp macro library and store it as a project macro library, just so that I would have control over it.

[Tom]

In general what I have used is hierarchy of autocall macros.  In the SASAUTOS option you can specify the order that they are searched.  I have normally used a local first search order (reporting event, subject area, global, sas supplied) which allows for subject areas or reporting events to override the definition that lives at a higher level.  So you can make better versions of some of the SAS supplied macros if you wanted.  But you could make the argument to search in the opposite order so that overriding is not possible.

I have seen systems that require you to specify which macros you will use instead of using AUTOCALL to search for the macro definition.  For example one had a utility macro you would call telling it what macro you needed and it would find the source and %INCLUDE it instead.  Another one worked outside of SAS to use the metadata to populate a temporary directory the only the macros you said you needed and then run SAS using just that directory as the AUTOCALL location.  In these system if you forgot to "register" your use of the macro then SAS would generate an error when you tried to use it and your job would fail.

Both of those systems had a way of specifying a version to use instead of just the default most recent version.  One was driven only by DATETIME stamp.  So "give the the versions of macro X, Y and Z that was current on 01MAR2024:06:00".

[Quentin]

Thanks @Tom for your further thoughts.  I'm definitely a fan of using a hierarchy of autocall macro libraries.  I also search the local first, so I search project macros, then corporate macros, then SAS-provided macros.  I think it's good to allow the over-rides.  For some projects my project macro directory may be empty.  But if for some reason I need to tweak a corporate macro, I can put it in the project directory and update it.

I hadn't thought about a system where you need to "register" macros before they are used, rather than use an actual autocall library.  That's an interesting idea. For my first few years of using macros, I used a %INCLUDE approach for my personal macro library, but it didn't have a dynamic feature to specify a date/version.

In fact that reminds that @KevinViel maybe consider storing the autocall macros in a SAS package https://github.com/yabwon/SAS_PACKAGES ? That would also give users an easy path to "install" on demand, and the ability to specify the specific version to be used. I'm sure @yabwon would be happy to talk you through that approach.

[yabwon]

SAS Packages have "version control" in their design. 

When loading a package to your SAS Session you can force that only specyfic version will be allowed to load, or "a version at least as high as required",

Furthermore, package developer sets the order of loading.

Packages can be stored in multiple directories and the order of searching a package to load can be set.

Plus since packages have both version and timestamp of creation imprinted in them, there are at least two ways to verify if this is the one we need. Plus the %verifyPackage() macro allows to compare check sum (hash digest) provided by the developer with current file, so we can be sure that files with code weren't modified somewhere along the way.

Further more, during a package generation process (when proper option is enabled) both "packageName.zip" and "packageName_version_.zip" are generated, so you won't overwrite historical version that easy.

Features are there, tool is there, also a bunch of tutorials/videos are available, and, as @Quentin wrote, I'll be more than happy to talk about it 🙂 

All the best

Bart

[KevinViel]

We have none to my knowledge.  I had written a suite of macros to be a version control system since storing whole versioned SAS programs in a SAS data set even with 20-100 versions (updates) would be miniscule, per se.  It would be like Razor, reporting the versions in DEV, QA, and PROD.   My macros have not been reviewed and certainly not Validated.

Thank you,

Kevin

[Tom]

We used Razor for version control on SAS environment/system we made at the end of the last century.  Had a SAS macro so users could ci/co/promote files.

The system had both hierarchical structure and DEV/QC/PROD states.   Standard libnames for data and/or metadata would be pointed to the right location based both on project you were using and what state level you wanted to use.  So if you point to data to DEV then you could read/write data into the DEV directory, but only read data from the PROD directory.  That way you could test you program and then when you promote it there is no change other than the environment that it runs in.

Example initialization report:

--------------------------------------------------------------------------
Current Environment: source project global global project raw   scrf  va
     hostname        code   macros  macros meta   meta    data  data  data
     9.04.01M7       -----------------------------------------------------
 07NOV2024:17:59:41  prod   prod    prod   prod   prod    prod  prod  prod
--------------------------------------------------------------------------

I still have SAS macros for RCS ci/co commands that work in Display Manager on Unix.  Including versions that work from the SAS command line to ci and/or co the current program you are editing.  So for example from the command prompt in the Editor window I could modify a program from the global macro library using a sequence like:

e gmac init.sas
co
w
ci Reason for change goes here

So between the E command to start editing the file you could make you changes and test your program. Before you could write changes you would need the co command to check out (default is the current file you are editing).  The w command will save the current file.  SAS's normal prompt showing the actual file name and asking for confirmation will appear every time.  Finally you can issue the ci command to check in the new version.  The rest of the command line after the ci command is the reason that will stored in the RCS file for this change.

[KevinViel]

"We used Razor for version control on SAS environment/system we made at the end of the last century." 😲😁😁

I think version control is somewhat underappreciated in this field.  I was assigned to one company that used it and I would say that they had one of the best systems (processes) that I have since in about 15 years in this field (the beginning of the current century 😆).  I need to think about it more.

I liked Razor.  I like my directory-level audits/reconciliations.  I treat every project as if 1) I would give the treatment to my infant child or mother and 2) as if it were my discovery on its make-or-break path from the bench to the bedside.

I have certainly been helped, like many others, by your posts and insights.

Best regards,

Kevin