cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
BookmarkSubscribeRSS Feed

Accessing AWS S3 as NFS from CAS and SAS – Part 3

Started ‎06-06-2019 by
Modified ‎06-07-2019 by
Views 2,065

In Part 1 of this article series, I talked about mounting an S3 bucket to CAS and SAS servers as a Network File System. Mounting an AWS S3 bucket as a filesystem means you can use all your existing tools to interact with S3 bucket to perform read and write operations on files and folders. This article is about another method “Objective-FS” to mount S3 as NFS. With Objective-FS software installed on to CAS and SAS server you can load data files from S3 bucket to CAS, save a CAS table to S3 bucket, and access S3 data files in base SAS.

Objective FS:

Objective-FS is a shared distributed POSIX file system that provides persistent data storage among the cloud instance, local machine and on-premise servers. The Objective-FS software runs on the on-premise server and talks directly to the cloud object store using S3 API. It supports multiple writers and readers; multiple nodes can write to the same file system concurrently and see each other’s data. The speed and performance of filesystem are maintained by using workload heuristics such as write bundling and read caching. The Object-FS uses cloud secure end-to-end client crypto to read and write data to S3. The data is encrypted at rest and in motion. The data is encrypted locally with key (known to a user only) before it leaves the on-premise data server.

 

The software cost!! The price for the professional version software is $49/- per month. The price for the enterprise version software is $299/- per month. For more detail please visit vendor’s product price page.

 

The following steps describe how to install, configure and use the Objective-FS at SAS and CAS server to access the S3 bucket:

  • Create an account at Obectivefs.com.

    To use the Object-FS software you need to have an active account at objectivefs.com. The user profile contains the product license and authorization key, which is required during objective-FS server configuration at SAS and CAS server. The download page contains the link to download the required software for the respective OS. The following screenshots describe the Objective-FS license and download page.

     

    S3FS_NFS_P3_Pic1.png

    Select any image to see a larger version.
    Mobile users: To view the images, select the "Full" version at the bottom of the page.

     

    S3FS_NFS_P3_Pic2.png

     

  • Install Objective-FS software on CAS and SAS servers.

    Once you have downloaded the required software for the respective OS, sftp it to the desired server for software installation. The software installation must be executed as user ‘root’ or user with ‘sudo’ access. The following step describes the objective-FS software installation at CAS and SAS server. You also need to verify hosting server NTP for small offset.

     

    ## At CAS Server 

[root@intcas01 ~]# ls -l obj*
-rw-r--r--. 1 root root 240984 Apr 3 15:48 objectivefs-6.1-1.x86_64.rpm
[root@intcas01 ~]#

 
[root@intcas01 ~]# yum install objectivefs-6.1-1.x86_64.rpm

Examining objectivefs-6.1-1.x86_64.rpm: objectivefs-6.1-1.x86_64
Marking objectivefs-6.1-1.x86_64.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package objectivefs.x86_64 0:6.1-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=================================================================================================================================================
 Package                          Arch                        Version                       Repository                                      Size
=================================================================================================================================================
Installing:
 objectivefs                      x86_64                      6.1-1                         /objectivefs-6.1-1.x86_64                      507 k

Transaction Summary
=================================================================================================================================================
Install  1 Package

Total size: 507 k
Installed size: 507 k
Is this ok [y/d/N]:  y 
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : objectivefs-6.1-1.x86_64                                                                                                      1/1
  Verifying  : objectivefs-6.1-1.x86_64                                                                                                      1/1

Installed:
  objectivefs.x86_64 0:6.1-1

Complete!
[root@intcas01 ~]#

[root@intcas01 ~]# /usr/sbin/ntpdate -q pool.ntp.org

server 198.60.22.240, stratum 0, offset 0.000000, delay 0.00000
server 4.53.160.75, stratum 0, offset 0.000000, delay 0.00000
server 104.168.88.15, stratum 0, offset 0.000000, delay 0.00000
server 69.164.213.136, stratum 0, offset 0.000000, delay 0.00000
 3 Apr 16:06:17 ntpdate[20764]: no server suitable for synchronization found
[root@intcas01 ~]#

     

    ## At SAS/Viya Server 

[root@intviya01 ~]# ls -l obj*
-rw-r--r--. 1 root root 240984 Apr  3 15:57 objectivefs-6.1-1.x86_64.rpm
[root@intviya01 ~]#

[root@intviya01 ~]# yum install objectivefs-6.1-1.x86_64.rpm

Examining objectivefs-6.1-1.x86_64.rpm: objectivefs-6.1-1.x86_64
Marking objectivefs-6.1-1.x86_64.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package objectivefs.x86_64 0:6.1-1 will be installed
--> Finished Dependency Resolution
…..
……………

Dependencies Resolved

=================================================================================================================================================
 Package                          Arch                        Version                       Repository                                      Size
=================================================================================================================================================
Installing:
 objectivefs                      x86_64                      6.1-1                         /objectivefs-6.1-1.x86_64                      507 k

Transaction Summary
=================================================================================================================================================
Install  1 Package

Total size: 507 k
Installed size: 507 k
Is this ok [y/d/N]: y 
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : objectivefs-6.1-1.x86_64                                                                                                      1/1
  Verifying  : objectivefs-6.1-1.x86_64                                                                                                      1/1

Installed:
  objectivefs.x86_64 0:6.1-1

Complete!
[root@intviya01 ~]#

[root@intviya01 ~]# /usr/sbin/ntpdate -q pool.ntp.org

server 69.164.213.136, stratum 0, offset 0.000000, delay 0.00000
server 4.53.160.75, stratum 0, offset 0.000000, delay 0.00000
server 198.60.22.240, stratum 0, offset 0.000000, delay 0.00000
server 104.168.88.15, stratum 0, offset 0.000000, delay 0.00000
 3 Apr 16:06:04 ntpdate[9945]: no server suitable for synchronization found
[root@intviya01 ~]#

     

  • Configure the Objective-FS server credentials to access S3 bucket from CAS and SAS servers.

    To configure objective-FS server a valid AWS_ACCESS_KEY_ID and AWS_SECRET_ACECSS_KEY are required from AWS account. The ACCESS_KEY and SECRET_KEY are used for accessing S3 location. The process to get the ACCESS_KEY and SECRET_KEY is explained in a previous post. The following step describes the S3 access credential configuration for Objective-FS server on CAS and SAS server.

     

    ## On CAS Server 

[root@intcas01 ~]# mount.objectivefs config 
Creating config in /etc/objectivefs.env
Enter ObjectiveFS license: XXXXXXXXXXXXXXXXXXXXXXXX
Enter Access Key Id: XXXXXXXXXXXXXXXXXXX
Enter Secret Access Key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Enter Default Region (optional): us-east-1
Creating:
  /etc/objectivefs.env
  /etc/objectivefs.env/OBJECTIVEFS_LICENSE
  /etc/objectivefs.env/AWS_ACCESS_KEY_ID
  /etc/objectivefs.env/AWS_SECRET_ACCESS_KEY
ObjectiveFS was successfully configured.
[root@intcas01 ~]#

     

    ## On SAS/Viya Server 

[root@intviya01 ~]#  mount.objectivefs config 
Creating config in /etc/objectivefs.env
Enter ObjectiveFS license: XXXXXXXXXXXXXXXXXXXXXXXX
Enter Access Key Id: XXXXXXXXXXXXXXXXXXX
Enter Secret Access Key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Enter Default Region (optional): us-east-1
Creating:
  /etc/objectivefs.env
  /etc/objectivefs.env/OBJECTIVEFS_LICENSE
  /etc/objectivefs.env/AWS_ACCESS_KEY_ID
  /etc/objectivefs.env/AWS_SECRET_ACCESS_KEY
ObjectiveFS was successfully configured.
[root@intviya01 ~]#

     

  • Create an objective-FS filesystem/bucket in the default S3 location region.

    The objective-FS creates its own S3 bucket to store multiple fragmented data files. You cannot use an existing bucket with Objective-FS environment. The passphrase created at this step is used when mounting a local folder to S3 location. Please keep the passphrase in a safe and secure place, without passphrase you can not access the files from S3 bucket created in this step. All files and objects at S3 are distributed and encrypted by Objective-FS. There will be multiple encrypted files for one data file.

     

    [root@intcas01 ~]#   mount.objectivefs create -l us-east-1 s3://gelsasfs 
Passphrase (for s3://gelsasfs): 
Verify passphrase (for s3://gelsasfs): 
NOTE: 3 PUT, 2 LIST, 0 GET, 0 DELETE, 2.491 kB IN, 2.696 kB OUT, CACHE [0.0% HIT, 0.0% META, 0.0% DATA, 704.0  B OS], DISK [0.0% HIT]
NOTE: Successfully created filesystem: s3://gelsasfs in the us-east-1 region
[root@intcas01 ~]#

     

    The bucket created from the previous steps can be viewed at the S3 user interface. Notice the files created under this bucket is encrypted and can only be read and accessed by objective-FS mounting point.

     

    S3FS_NFS_P3_Pic3.png

     

    S3FS_NFS_P3_Pic4.png

     

  • Mount a local folder at CAS and SAS server to Objective-FS file system/bucket.

    To mount the objective-FS file system, you need an existing empty local folder. The Objective-FS process will run in background for the mounting point. When mounting the filesystem it will ask the passphrase created from previous (mount.objectivefs create) step.

     

    ## On CAS server 
[root@intcas01 ~]# mkdir /opt/sas/s3objfsmnt
[root@intcas01 ~]# chown sas:sas /opt/sas/s3objfsmnt
[root@intcas01 ~]# chmod 777 /opt/sas/s3objfsmnt
[root@intcas01 ~]# mount.objectivefs gelsasfs /opt/sas/s3objfsmnt
Passphrase (for s3://gelsasfs): [ XXXXXXXXX  ](Passphrase created from mount.objectivefs create step)
[root@intcas01 ~]#

     

    ## On SAS / Viya Server 
[root@intviya01 ~]# mkdir /opt/sas/s3objfsmnt
[root@intviya01 ~]# chown sas:sas /opt/sas/s3objfsmnt
[root@intviya01 ~]# chmod 777 /opt/sas/s3objfsmnt
[root@intviya01 ~]# mount.objectivefs gelsasfs /opt/sas/s3objfsmnt
Passphrase (for s3://gelsasfs): [ XXXXXXXXX] (Passphrase created from mount.objectivefs create step)
[root@intviya01 ~]#

     

  • Verify the Objective-FS mounted filesystem at CAS and SAS/Viya server.

    The following step describes the verification of objective-FS mounting point, data file copy from local folder to S3 mounting point and data files accessible from both CAS and SAS servers.

     

    ## At CAS Server 

## list objective-FS mounting to S3 bucket
[root@intcas01 ~]# mount.objectivefs list  
NAME                    KIND    SNAP    REGION          LOCATION
s3://gelsasfs           ofs     on      us-east-1       US East (N. Virginia)

## list files from mounting folder
[root@intcas01 ~]# ls -l /opt/sas/s3objfsmnt
total 1
-rw-rw-rw-. 1 root root 605 Apr  4 11:57 README
[root@intcas01 ~]#

## Copy data files to Objective-FS file system/fodler. 
[root@intcas01 ~]# cp /opt/sas/data/customers.sas7bdat /opt/sas/s3objfsmnt/
[root@intcas01 ~]# cp /opt/sas/data/prdsale.sas7bdat  /opt/sas/s3objfsmnt/

## list files from mounting folder
[root@intcas01 ~]# ls -l /opt/sas/s3objfsmnt
total 1153
-rw-r--r--. 1 root root 917504 Apr  4 12:10 customers.sas7bdat
-rw-r--r--. 1 root root 262144 Apr  4 12:10 prdsale.sas7bdat
-rw-rw-rw-. 1 root root    605 Apr  4 11:57 README
[root@intcas01 ~]#

[root@intcas01 ~]# df -k
Filesystem                                                1K-blocks       Used        Available Use% Mounted on
/dev/mapper/rhel-root                                     200671504  106605168         94066336  54% /
devtmpfs                                                   16380036          0         16380036   0% /dev
tmpfs                                                      16391040         76         16390964   1% /dev/shm
tmpfs                                                      16391040    1647520         14743520  11% /run
tmpfs                                                      16391040          0         16391040   0% 
s3://gelsasfs                                      1125899906842624        312 1125899906842312   1% /opt/sas/s3objfsmnt
[root@intcas01 ~]#

     

    ## At SAS / Viya Server 

## list objective-FS mounting to S3 bucket
[root@intviya01 ~]$ sudo mount.objectivefs list 
NAME                    KIND    SNAP    REGION          LOCATION
s3://gelsasfs           ofs     on      us-east-1       US East (N. Virginia)

## list files from mounting folder
[root@intviya01 ~]# ls -l /opt/sas/s3objfsmnt
total 1153
-rw-r--r--. 1 root root 917504 Apr  4 12:10 customers.sas7bdat 
-rw-r--r--. 1 root root 262144 Apr  4 12:10 prdsale.sas7bdat 
-rw-rw-rw-. 1 root root    605 Apr  4 11:57 README
[root@intviya01 ~]#

[root@intviya01 ~]# df -k
Filesystem                                                1K-blocks       Used        Available Use% Mounted on
/dev/mapper/rhel-root                                     200671504   74814312        125857192  38% /
devtmpfs                                                   49407664          0         49407664   0% /dev
tmpfs                                                      49418664         24         49418640   1% /dev/shm
tmpfs                                                      49418664    4155988         45262676   9% /run
tmpfs                                                      49418664          0         49418664   0% /sys/fs/cgroup
/dev/sda1                                                    508588     221964           286624  44% /boot
s3://gelsasfs                                      1125899906842624        312 1125899906842312   1% /opt/sas/s3objfsmnt
[root@intviya01 ~]#

     

    The data files at the S3 location are fragmented and encrypted. It can only be read via the objective-FS filesystem.

     

    S3FS_NFS_P3_Pic5.png

     

  • Objective-FS Process running on OS.

    There is an OS process to support the access of S3 bucket from Objective-FS. After OS reboot, if this process is down, you need to mount the filesystem again using the previous step.

     

    [root@intcas01 ~]# ps -eaf | grep objective
root     21862     1  2 12:03 ?        00:03:19 mount.objectivefs gelsasfs /opt/sas/s3objfsmnt 
root     27614 12143  0 14:21 pts/1    00:00:00 grep --color=auto objective
[root@intcas01 ~]#

     

  • Objective-FS S3 Bucket data file access from BASE SAS.

    After mounting objective-FS S3 bucket to SAS compute server, the data files can be accessed from BASE SAS using LIBNAME and FILENAME statement. The following example describes the S3 bucket data file access from base SAS using a LIBNAME statement.

     

    libname mylib "/opt/sas/s3objfsmnt " ;
proc print data=mylib.customers ; 
run;

     

    S3FS_NFS_P3_Pic6.png

     

  • Objective-FS S3 Bucket data file access from CAS.

    After mounting objective-FS S3 bucket to CAS Controller, the data files can be accessed using path-based CASLIB. The following code example describes the data load to CAS from an S3 bucket and data save from CAS to an S3 bucket via objective-FS route. Other SAS application can also use the newly saved .sas7bdat and .csv files via objective-FS mount.

     

    CAS mySession  SESSOPTS=( CASLIB=casuser TIMEOUT=99 LOCALE="en_US");
caslib caslibs3 datasource=(srctype="path")  path="/opt/sas/s3objfsmnt" ; 

/* load a S3 data file to CAS */ 
PROC CASUTIL incaslib="caslibs3" outcaslib="caslibs3";
    droptable casdata="prdsale"  quiet; 
	LOAD casdata="prdsale.sas7bdat" CASOUT="prdsale" copies=0
   importoptions=(filetype="basesas", dtm="auto", debug="dmsglvli"); 
RUN;
quit;

/* Save a CAS table to S3 with .sashdat extension */ 
proc casutil  incaslib="caslibs3" outcaslib="caslibs3";
   save casdata="prdsale"  casout= "prdsale_new" replace ;
run;
quit;

/* Save a CAS table to S3 with .sas7bdat extension */ 
proc casutil  incaslib="caslibs3" outcaslib="caslibs3";
   save casdata="prdsale"  casout= "prdsale_new.sas7bdat" replace ;
run;
quit;

/* Save a CAS table to S3 with .csv extension */ 
proc casutil  incaslib="caslibs3" outcaslib="caslibs3";
   save casdata="prdsale"  casout= "prdsale_new.csv" replace ;
run;
quit;

/* load a .sashdat file from S3 to CAS */ 
proc casutil incaslib="caslibs3" outcaslib="caslibs3";
    droptable casdata="prdsale_new_hdat" quiet;
	load casdata="prdsale_new.sashdat" casout="prdsale_new_hdat"  ; 
run;
quit;

/* load a .csv file from S3 to CAS */ 
proc casutil incaslib="caslibs3" outcaslib="caslibs3";
    droptable casdata="prdsale_new_csv" quiet;
	load casdata="prdsale_new.csv" casout="prdsale_new_csv"  ; 
run;
quit;

proc casutil;
        list tables incaslib="caslibs3 ";
        list files incaslib="caslibs3 ";
run;

/* Shutdown CAS Session */
CAS mySession  TERMINATE;

     

    Result extracts from the above code execution.

     

    S3FS_NFS_P3_Pic7.png

     

    S3FS_NFS_P3_Pic8.png

     

    The newly saved data files from CAS to S3 are also accessible from BASE SAS.

     

    S3FS_NFS_P3_Pic9.png

     

Data load Performance from S3 bucket to CAS using objective-FS:

The data load to CAS from S3 bucket using objective-FS S3 filesystem is slow compared to data load from the local filesystem. The parallel load using objective-FS S3 mounted file system is not supported so the load time is listed using serial mode. S3 Type CAS lib only support SASHDAT and CSV file format, so for better comparison following test was conducted using a .sashdat file.

 

Note: The SAS Viya 3.5 is coming with improved performance when loading data from S3 to CAS using S3 type CASLIB.

 

S3FS_NFS_P3_Pic10.png

Summary:

  • The Objective-FS filesystem enables you to access the S3 bucket data file from CAS and SAS servers.
  • The data files are encrypted in transit and at rest.
  • The data file stored at the S3 location using objective-FS can read by only objective-FS mounting.

Important link: Objective-FS

Version history
Last update:
‎06-07-2019 07:50 AM
Updated by:
Community Manager

sas-innovate-2024.png

Don't miss out on SAS Innovate - Register now for the FREE Livestream!

Can't make it to Vegas? No problem! Watch our general sessions LIVE or on-demand starting April 17th. Hear from SAS execs, best-selling author Adam Grant, Hot Ones host Sean Evans, top tech journalist Kara Swisher, AI expert Cassie Kozyrkov, and the mind-blowing dance crew iLuminate! Plus, get access to over 20 breakout sessions.

 

Register now!

Free course: Data Literacy Essentials

Data Literacy is for all, even absolute beginners. Jump on board with this free e-learning  and boost your career prospects.

Get Started

Article Labels
Article Tags