topic Re: Sourcing in (macro) code from a _private_ repository in SAS Programming

Sourcing in (macro) code from a _private_ repository

fja — Tue, 22 Aug 2023 08:47:11 GMT

Dear Community!

I am looking for a way to %include code from a github repository (several autocall macros). I can do so with a public repository, but what about a private repository?

We have had several discussions here on accessing git repositories directly and reading data from it (both data and code or just macro code). But all these approaches require a public repository to have the links to the data/code code working (equivalent to the raw file button in github). In this posting @ChrisHemedinger has sketched out what I actually want to achive (first code fragment) ... and what I would like to avoid (second code fragment using the gitfn_*).

The problem is ... that the link to the raw file in a private repository gets an oauth token appended, but that changes over time. And private access tokens cannot be used in the same fashion (at least by the looks of it). Instead the wise internet suggest the use of a header ("Authentication: token TOKEN") for a similar access problem using curl. In curl there is a nice option to do this ... but how to do this in SAS?

Just another piece of information: I am limited to the use of GitHub Enterprise and non public repositories. It could very well be that I am barking up the wrong tree ... maybe ssl authentication is the way to go. So, to cut it short:

Does anybody have any experience with accessing non public github repositories under SAS and using %include to source in autocall macros? (and is willing to share his expertise of course 😃)

Regards

fja

Re: Sourcing in (macro) code from a _private_ repository

Reeza — Tue, 22 Aug 2023 15:36:41 GMT

What version of SAS (Studio/EG/Viya)?

Are you looking at connecting to your repository overall or rather just at startup?

Re: Sourcing in (macro) code from a _private_ repository

fja — Tue, 22 Aug 2023 18:48:22 GMT

Hello Reya!

Thank you very much for your reply.

SAS 9.4 EG 8.2, but using %include should be independant of the EG version.

It is just a punctual connect i am looking at just for establishing a connection via filename ... to a single file in a repository or a directory.

The first shall be used for sourcing in a single file ... the latter would be used to set up the autosas option.

Cheers
Fja

Re: Sourcing in (macro) code from a _private_ repository

Reeza — Tue, 22 Aug 2023 18:50:07 GMT

Right, if you don't want to connect and create a local repository, it does make it a more complicated process. Especially if doing this for multiple files.

Re: Sourcing in (macro) code from a _private_ repository

fja — Tue, 22 Aug 2023 20:06:35 GMT

I feared it to be so ... nevertheless from the coding side it is ever so charming: setting up a filename and off you go ...
But you are correct, doing it via a temporary clone is doable ... but then again one needed to ensure that your local copy (clone) is always updated to the latest ...

Re: Sourcing in (macro) code from a _private_ repository

Reeza — Tue, 22 Aug 2023 21:14:45 GMT

Yes, but git functions are smart enough to only update files as needed.

Re: Sourcing in (macro) code from a _private_ repository

Quentin — Tue, 22 Aug 2023 21:22:38 GMT

@Reeza wrote:
Yes, but git functions are smart enough to only update files as needed.

Good point. So presumably you could wrap Chris's code from the other thread in an autocall macro. So user could run %gitinclude(repo=,file=) and it would use SAS git functions to update the local repo, then %include the specified .sas file. I suppose if you wanted to, the local repo could even be put in the WORK dir.

Re: Sourcing in (macro) code from a _private_ repository

Reeza — Tue, 22 Aug 2023 21:30:46 GMT

If you use work, can use gitclone and then a %include, I would wrap the whole thing in a local macro and call that?

Re: Sourcing in (macro) code from a _private_ repository

fja — Thu, 24 Aug 2023 14:44:59 GMT

NOTE: libgit2.so not found in the system location, trying the extension path.
NOTE: libgit2.so not found in the system location, trying the install path.
ERROR: libgit2.so not found.
ERROR: Unable to load libgit2 module.

Lovely! So I need to check with the admins first ... It might not be a wandering star I was born under ... but it is not a good one by the looks of it. 🙂

Re: Sourcing in (macro) code from a _private_ repository

Reeza — Thu, 24 Aug 2023 14:48:31 GMT

Are you on 9.4M6+?

Re: Sourcing in (macro) code from a _private_ repository

fja — Thu, 24 Aug 2023 15:12:50 GMT

Current version: 9.04.01M7P080520

i.e. yes. 😀

Re: Sourcing in (macro) code from a _private_ repository

fja — Fri, 25 Aug 2023 07:51:02 GMT

Ok, admins say they need a concept for using git and "if and how" git could be integrated with SAS. I read that as "go away". 😃

Question: Does anybody know, if SAS brings its own git (i.e. the shared objects) or does it rely on the OS to provide these? I was under the impression that the first case held true.

Re: Sourcing in (macro) code from a _private_ repository

yabwon — Fri, 25 Aug 2023 09:44:07 GMT

If the private repo is owned by your team (e.g. user MySuperTeamRepo) you could try to generate Private Access Token for a repo:

https://docs.github.com/en/enterprise-server@3.6/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-personal-access-token

Settings:

https://github.com/settings/tokens

token type: classic

with the scopes:

repo - Full control of private repositories
repo:status - Access commit status
repo_deployment - Access deployment status
public_repo - Access public repositories
repo:invite - Access repository invitations
security_events - Read and write security events

and then:

filename x url 'https://raw.githubusercontent.com/MySuperTeamRepo/REPO_NAME/main/FILE_TO_INCLUDE.sas' 
user = 'MySuperTeamRepo' 
password = 'ghp_JKFGHKLDRF*********GHFKJDBVKJSDVJKHSDJK' /* <- token goes here */
/*debug*/
;

%include x / source2;

You can parameterise it with macro variables:

[EDIT: semicolon was missing in macro definition]

%macro GetMeTheCode(
/* required */
 user
,token
,repoName
,file
/* optional */
,branch=main
,debug= /*debug*/
,source2= /*source2*/
);

filename x url "https://raw.githubusercontent.com/&user./&repoName./&branch./&file." 
user = "&user." 
password = "&token." /* <- token goes here */
&debug.
;
filename x list;

%include x / &source2.;

filename x clear;

%mend GetMeTheCode;


%GetMeTheCode(
 MySuperTeamRepo
,ghp_JKFGHKLDRF*********GHFKJDBVKJSDVJKHSDJK
,REPO_NAME
,FILE_TO_INCLUDE.sas
)

It's not a silver bullet since the token allows access to all repositories owned by MySuperTeamRepo, but maybe it will do the job for you.

All the best

Bart

Re: Sourcing in (macro) code from a _private_ repository

yabwon — Fri, 25 Aug 2023 09:41:48 GMT

If you decide to set up the "fine-grained token" with parameters:

Only select repositories: your selected repo

and

Repository permissions: Contents (Repository contents, commits, branches, downloads, releases, and merges.) in Read_Only

that should give you some better "access" setup.

Bart

Re: Sourcing in (macro) code from a _private_ repository

fja — Fri, 25 Aug 2023 09:51:03 GMT

@yabwon wrote:

token type: classic

with the scopes:

repo - Full control of private repositories
repo:status - Access commit status
repo_deployment - Access deployment status
public_repo - Access public repositories
repo:invite - Access repository invitations
security_events - Read and write security events

and then:
filename x url 'https://raw.githubusercontent.com/MySuperTeamRepo/REPO_NAME/main/FILE_TO_INCLUDE.sas' 
user = 'MySuperTeamRepo' 
password = 'ghp_JKFGHKLDRF*********GHFKJDBVKJSDVJKHSDJK' /* <- token goes here */
/*debug*/
;

%include x / source2;

Hello Bart!

Thank you for your reply. This is what I did when first trying it ... the filename line fails on me with a timeout ... though the host is pingable.

Just that I am using (have to use) GitHup Enterprise.

Cheers

PS: And of course I did retry it with the code you suggested ... same result: timeout.

Re: Sourcing in (macro) code from a _private_ repository

yabwon — Fri, 25 Aug 2023 10:26:34 GMT

Hmmm...

I did a test with a "organisation" repo I have and it worked too. But "organiosation" repo it is not the same a "enterprise"... maybe some different setup is needed...

[EDIT:] maybe option: add a collaborator with read access only will do? but this will require that all possible users would have git account too...

Bart

Re: Sourcing in (macro) code from a _private_ repository

fja — Fri, 25 Aug 2023 11:54:35 GMT

Another piece of information:

when using x command git clone git clone https;//<MeName>:<MePwd>@<host>/<MeName>/<repo>.git on the sas host I get

Cloning into '<repo>'...
/usr/lib/git/git-remote-https: /opt/sas/sas94m7/SASHome/SASFoundation/9.4/sasexe/libcrypto.so.1.0.0: no version information available (required by /usr/lib64/libcurl.so.4)
/usr/lib/git/git-remote-https: /opt/sas/sas94m7/SASHome/SASFoundation/9.4/sasexe/libssl.so.1.0.0: no version information available (required by /usr/lib64/libcurl.so.4)
/usr/lib/git/git-remote-https: /opt/sas/sas94m7/SASHome/SASFoundation/9.4/sasexe/libcrypto.so.1.0.0: no version information available (required by /usr/lib64/libssh.so.4)
/usr/lib/git/git-remote-https: /opt/sas/sas94m7/SASHome/SASFoundation/9.4/sasexe/libssl.so.1.0.0: no version information available (required by /usr/lib64/libldap-2.4.so.2)
/usr/lib/git/git-remote-https: /opt/sas/sas94m7/SASHome/SASFoundation/9.4/sasexe/libcrypto.so.1.0.0: no version information available (required by /usr/lib64/libldap-2.4.so.2)
fatal: unable to access 'https://<host>/<MeName>/<repo>.git/': Failed to connect to <host> port 443 after 7247 ms: Couldn't connect to server

Funny/remarkable thing is: the unix git command (/usr/bin/git) draws the sas so-libs.

This seems to indicate that the git setup is messy on the host. EG however lets me clone it to the local PC, so the credentials seem to work.

Re: Sourcing in (macro) code from a _private_ repository

fja — Fri, 25 Aug 2023 11:57:51 GMT

@ Bart: Thank you for indicating how it works on your side ... I guess this should be the solution I am looking for iff the git setup was correct. (see my posting above)

Re: Sourcing in (macro) code from a _private_ repository

AhmedAl_Attar — Fri, 25 Aug 2023 12:45:02 GMT

Hi @fja

I have a feeling your System Admins would need to install git on your SAS server before you could SAS's git functions (gitfn_*)

Hope this helps

Re: Sourcing in (macro) code from a _private_ repository

Quentin — Fri, 25 Aug 2023 13:24:44 GMT

@fja wrote:

Ok, admins say they need a concept for using git and "if and how" git could be integrated with SAS. I read that as "go away". 😃

Question: Does anybody know, if SAS brings its own git (i.e. the shared objects) or does it rely on the OS to provide these? I was under the impression that the first case held true.

I *think* you are correct that SAS brings its own git. Pretty sure you can use EG or Studio to connect to git, and also use the git functions in SAS code, without ever installing git on the OS.

But in my limited experience, I think it would be hard to use git without having it also available on the OS so you can use non-SAS git interfaces/clients. We store our SAS code on a linux server, and have central git repos on a Windows server somewhere. I tried playing with git, but the admins weren't interested in investing time to get the authentication working between our linux SAS box and the windows box with the git repo. I don't think I could have done it from within SAS even if I had tried. I have played with some ugly workarounds running git locally on windows, so I can connect to the central git repos from windows, and accessing the linux files via samba....