For better Git usage: Save all code elements in an Enterprise Guide project externally

[GrandGrue]

We use Git and GitLab for all our code (Python, R). In order to make more intensive use of version control for SAS code and projects that were created with Enterprise Guide, the self-written SAS code would have to be stored in external files. SEG normally saves the code in the EGP file. This means that it is hidden in a binary file and not visible to Git. Enterprise Guide offers the option of saving a code element individually externally. A relative path to the project file can be used. This option is good, but especially in large EG projects it is tedious to manually save each individual code element externally and to ensure that no code element is stored internally. For this reason, we are looking for a script or similar that can support us in this work. Has anyone already done something similar?

[dalocja]

To automate saving SAS code elements from Enterprise Guide to external files for better Git integration, consider using a Python or SAS script that iterates through all code nodes in the project and exports them to specified paths.

[GrandGrue]

Thank you for this suggestion. I already had that idea too. Unfortunately, I could not find an online source that describes all the classes and functions that are necessary for this. Everything I have found so far refers to EG 4.1, which was a code base that did not yet allow referential storage in projects, for example (only came with EG 4.3). Do you know a description of the scripting library which is up to date (EG 8)?

[AlanC]

EGP files are merely binary zip files. They can be unzipped and the XML extracted. Playing version control that way, however, would be dangerous. That said, try it and see how it goes. Rename the file from ex. MyFile.egp to MyFile.zip and you can explore the structure.

Using C#, I have unwound the ZIP files as well as the XML so I can extract out the actual SAS code. That app is on github under SaviApp. Note: the EGP XML files and how everything holds together took a lot of work to figure out. Be prepared if you want to figure it out in python..  

[GrandGrue]

Thank you very much for your support. This is my plan B if the above does not work. I have an almost finished solution with the following resource: https://github.com/cjdinger/sas-eg-automation/blob/master/vbscript/ExtractCodeAndLog.vbs
But I want to avoid this way, because changes to the code can only be mapped if the script is reliably executed before each commit.

[AlanC]

Let me save you some time (hopefully). The zip file changes each version (or major version). The XSD for the XML in those files is not public (SAS devs don't need it since they can direct map to the nodes). Think of the XML as the serialized version of the object. You have to guess at the deserialization.

The key, for you currently, is that it changes per version. This becomes more dramatic over time. VBS is also a poor language and very much a security risk. That might not matter in your situation. 

You can continue to press the VBS script but getting it to work (for more than the specific version it was built for) is probably futile and is a poor platform (vbs) to boot. YMMV but that is my .02 cents.

[GrandGrue]

Yes, I see the limitations. I even already have a piece of SAS code that can open the EGP/ZIP file and extract the code elements from it.
I would prefer to have a SAS option that always saves all code elements externally. Since this does not exist, I thought the second best option is a small script that does this. Once this is done, then code changes are always made in the external files and no post-processing is needed.

[AlanC]

Let me try and take a stab and how I would approach if given those circumstances. You can translate to python/your world and see if it makes sense.

I use C#/PowerShell so this is from that perspective.

• Create a PS FileSystemWatcher that watches a directory for a File Change. The directory is wherever user saves their projects (see one example: https://devblogs.microsoft.com/powershell-community/a-reusable-file-system-event-watcher-for-powersh...)
• Make that a Windows service so it monitors all of the time
• When that change is detected, fire a script to
  • Unwind the EGP
  • Save the code components to a .SAS file
  • Version control that SAS file

I am happy to share my C# code that unwinds the EGP and put it on github. Would have to put in a bit of time to get it ready for publication but I have no objections.  

[GrandGrue]

Thank you for the offer. I would be very interested in your code example.

[AlanC]

For now, use the command line tool that is there:

https://github.com/savian-net/SaviEg

[SASKiwi]

With EG 8 (and some earlier versions) you can completely bypass using EGPs and exclusively save your SAS code in external text files. This is what we do and it works fine with Git / Azure DevOps which we use for all of our production work.

[GrandGrue]

This is a valuable suggestion and we would also like to go down this route for new projects. My question concerns the approximately 1000 existing EG projects, which we would like to make more accessible for Git.
Let me illustrate my request better with screenshots (sorry that this is a German version 7.1):

• In the project settings, I can make EG save the path only relative when a file is saved externally (screenshot 01)
  
• If I use “Save as” in a code element, the element is saved externally (very good for Git / screenshot 02)
  
• The small arrow symbol shows code that is stored externally and only refers to it (screenshot 03)
  

I am now looking for a script that automates these manual “Save as” steps. In other words, I want to retain the EG's own functionality and not extract the code elements from the EGP.

[AlanC]

You can create a custom task using C#. EG is written in C# and it provides a means of doing this. You have to adhere to an interface and implement the parts.

There are examples on the SAS website and I should have an example or two on my github.  

[Patrick]

The Viya 4 SAS Studio git integration is imho much more streamlined. With the future in mind (it's actually already present) I'd be looking into the migration path of SAS EG to SAS Studio and the git integration capabilities of SAS Studio. 

The latest EG versions are now also compatible with SAS Viya but they don't offer the same richness in functionality. I'm especially fond of SAS Studio Flow and the growing number of SAS Studio Custom Steps available in public git repositories that one can use to build such flows.