how to specify wich .tks3.conf file to use with a pipe command

[alepage]

Hello,

I am using the following sas code to get the directory listing

%macro FindingMasterListPolicyName(env);

%let filelist=/&path1./s3filestodownload.txt;

/***** Selecting the good s3 bucket path according to the environment selected *****/

%if       &env eq dev  %then %let s3bucketpath=s3://dev-pr-brz-datareten-edp-nprod;
%else %if &env eq intg %then %let s3bucketpath=s3://intg-pr-brz-datareten-edp-nprod;
%else %if &env eq uat  %then %let s3bucketpath=s3://uat-pr-brz-datareten-edp-nprod;

%put &=env  &=s3bucketpath;
/*********************************************************************************/

filename oscmd pipe "aws s3 ls ""&s3bucketpath."" --recursive | grep .csv ";

data foldercontents2 ;
length text $2000. fnameandpath $200.;
infile oscmd;
input;
text=_infile_;
run;

data foldercontents2 (drop=text);
set foldercontents2 ;
creationdate=scan(text,1,' ');
creationtime=scan(text,2,' ');
filesize=scan(text,3,' ');
fnameandpath =scan(text,1," ","b");
FName =scan(text,1,"/","b");
path=substr(fnameandpath,1,length(fnameandpath)-length(FName)- 1);
where find(text,'MASTER_LIST') > 0 AND find(text,'_Policy_') > 0 AND find(text,'ARCHIVE') = 0;
run;

proc sort data=foldercontents2 out=foldercontents;
by creationdate creationtime FName;
run;


/*** Getting the path and file name of the Master List to download ***/
Data _null_;
set foldercontents;
if _n_ = 1  then 
do; 
	call symputx('MasterListName',Fname,'g');
    call symputx('masterlistfilepath',Path,'g');
end;
run;
%put &=masterlistfilepath. &=MasterListName.;

%goto exit;

filename filelist "&filelist";
data _null_;
FILE filelist LRECL=150 recfm=v;
put "&masterlistfilepath./&MasterListName.";
;
run;
%exit: %mend FindingMasterListPolicyName;
%FindingMasterListPolicyName(dev);

I have three conf file:

one in /home/dev/.tks3.conf

another one in / home/intg/tks3.conf

and another one in /home/uat/.tks3.conf

how to specify which one to use with the above pipe command ?

[Tom · Accepted solution]

Not really a SAS question.

You are using this operating system command to get the list of files.

aws s3

The AWS command has a command line option to specify which profile you want it to use.  From the command line on your host run this command

aws help

to see the help information for the aws command.  You will see:

      --profile (string)

       Use a specific profile from your credential file.

So create profiles in you credential file for your three environments. You might make profiles named with the same values as you pass in the ENV macro variable so that the only change you need to make is:

aws --profile &env ls 

[alepage]

filename oscmd pipe "aws s3 ls   &s3bucketpath. --profile &profile. --recursive | grep .csv ";

data foldercontents2 ;
length text $2000.;
infile oscmd;
input;
text=_infile_;
run;

You also need to go in your home directory /.aws/credentials

and in that credentials file, you add all your profiles as below:

[default]
aws_access_key_id = value
aws_secret_access_key = value
region = ca-central-1
[profile name1]
aws_access_key_id=value
aws_secret_access_key=value
region = ca-central-1
aws_session_token=value
[profile name2]
aws_access_key_id=value
aws_secret_access_key=value
region = ca-central-1
aws_session_token=value

[profile name3]
aws_access_key_id=value
aws_secret_access_key=value
region = ca-central-1
aws_session_token=value