Using SAS PROC S3 to access a bucket using CEPH

alisio_meneses — Thu, 25 Apr 2024 23:44:13 GMT

Hello There!

I'm having trouble connecting to a CEPH object storage service using PROC S3 in SAS Viya 3.5 on Linux. CEPH is an open-source distributed storage system capable of providing S3 compatible buckets. I've tried setting a custom region using the TKS3_CUSTOM_REGION environment variable and a config file with access credentials, but I keep getting an error saying "The AWS access key Id you provided does not exist in our records." Here's my code:
options set=TKS3_CUSTOM_REGION="us-east-1,storage.yada.yada,0,0,TRUE,TRUE";
%let myvar_value = %sysget(TKS3_CUSTOM_REGION);
%put &myvar_value;

PROC S3 
	config="/home/user/tks3.cfg";
	LIST "/";
RUN;
The S3 "/home/user/tks3.cfg" file contains the following:
region=us-east-1
keyID = thisIsAnAwesomeKeyID
secret = whoaThisSecretIsSafe
Output Log:
1    %studio_hide_wrapper;
83   options set=TKS3_CUSTOM_REGION="us-east-1,storage.yada.yada,0,0,TRUE,TRUE";
84   %let myvar_value = %sysget(TKS3_CUSTOM_REGION);
85   %put &myvar_value;
us-east-1,storage.yada.yada,0,0,TRUE,TRUE
86   
87   PROC S3
88   config="/home/user/tks3.cfg";
91   LIST "/";
92   RUN;
ERROR: Could not get user buckets.
ERROR: The AWS access key Id you provided does not exist in our records.
ERROR: The AWS access key Id you provided does not exist in our records.
NOTE: The SAS System stopped processing this step because of errors.
NOTE: PROCEDURE S3 used (Total process time):
      real time           0.58 seconds
      cpu time            0.02 seconds
      
93   
94   %studio_hide_wrapper;
105  
106  
Currently I am able to interact with data from the CEPH bucket using the command-line tool for managing Amazon S3 storage, s3cmd, the same way I use it to interact with AWS S3 buckets. I guess this rules out problems caused by the CEPH service.

Can someone point what might be wrong?

Thank you.

Environment info:

SAS Viya 3.5 running on Linux

Re: Using SAS PROC S3 to access a bucket using CEPH

alisio_meneses — Tue, 30 Apr 2024 12:44:35 GMT

I managed to resolve the issue by incorporating a custom region using a different approach. Since there was no response, I'll share my solution for anyone who might benefit from it.

Here's what I did to make it work:

Set a unique name for the custom region. Previously, I had assigned a custom region name identical to an existing AWS region, so I opted for a distinct name instead.
Utilized PROC S3 to introduce the custom region instead of relying on environment variables.

/*Add custom region*/

proc s3;
REGION ADD HOST="storage.middle.earth" NAME="shire";
run;
/* Listing all regions (including the custom region) */
proc s3;
region list;
run;

This resulted in the following log output (note the added custom region):

1 %studio_hide_wrapper;
83 proc s3;
84 REGION ADD HOST="storage.middle.earth" NAME="shire";
85 run;
NOTE: PROCEDURE S3 used (Total process time):
real time 0.01 seconds
cpu time 0.00 seconds

86
87 proc s3;
88 region list;
89 run;
Amazon Regions
us-east-1 s3.amazonaws.com Default 0
us-east-2 s3-us-east-2.amazonaws.com Default 0
us-west-2 s3-us-west-2.amazonaws.com Default 0
us-west-1 s3-us-west-1.amazonaws.com Default 0
eu-west-1 s3-eu-west-1.amazonaws.com Default 0
eu-central-1 s3-eu-central-1.amazonaws.com Default 0
ap-southeast-1 s3-ap-southeast-1.amazonaws.com Default 0
ap-southeast-2 s3-ap-southeast-2.amazonaws.com Default 0
ap-northeast-1 s3-ap-northeast-1.amazonaws.com Default 0
sa-east-1 s3-sa-east-1.amazonaws.com Default 0
us-gov-west-1 s3-us-gov-west-1.amazonaws.com Default 0
us-gov-west-1 s3-fips-us-gov-west-1.amazonaws.com Default 0
ca-central-1 s3-ca-central-1.amazonaws.com Default 0
ap-south-1 s3-ap-south-1.amazonaws.com Default 0
ap-northeast-2 s3-ap-northeast-2.amazonaws.com Default 0
cn-north-1 s3-cn-north-1.amazonaws.com Default 0
cn-northwest-1 s3-cn-northwest-1.amazonaws.com Default 0
eu-west-2 s3-eu-west-2.amazonaws.com Default 0
eu-west-3 s3-eu-west-3.amazonaws.com Default 0
ap-east-1 s3.ap-east-1.amazonaws.com Default 0
eu-south-1 s3.eu-south-1.amazonaws.com Default 0
eu-north-1 s3.eu-north-1.amazonaws.com Default 0
me-south-1 s3.me-south-1.amazonaws.com Default 0
af-south-1 s3.af-south-1.amazonaws.com Default 0
ap-northeast-3 s3.ap-northeast-3.amazonaws.com Default 0
Custom Regions
shire storage.middle.earth Default 0
NOTE: PROCEDURE S3 used (Total process time):
real time 0.00 seconds
cpu time 0.00 seconds

90
91 %studio_hide_wrapper;
102
103

Now, with the appropriate credentials, it's possible to test bi listing a bucket content:

proc s3 KEYID="isItSecret" SECRET="isItSafe" region="shire";
list "/bucket" ;
run;

That's it.

See ya!

topic Re: Using SAS PROC S3 to access a bucket using CEPH in SAS Procedures

Using SAS PROC S3 to access a bucket using CEPH

Re: Using SAS PROC S3 to access a bucket using CEPH