topic Re: Finding how much space is used in Hadoop using SAS EG in SAS Enterprise Guide

Finding how much space is used in Hadoop using SAS EG

amity21 — Sun, 21 Jul 2019 20:56:02 GMT

Hi,

We use SAS EG to run queries for our work. We have recently been provided a Hadoop location to save our datasets. I am looking for a way to capture the space used in that server/library.

I use to use the code

PROC SQL ;
create table SIZE_AC as 
SELECT LIBNAME,
MEMNAME,
FILESIZE FORMAT=SIZEKMG.,
FILESIZE FORMAT=SIZEK.
FROM DICTIONARY.TABLES
WHERE LIBNAME = 'AC'
AND MEMTYPE = 'DATA' 
ORDER BY FILESIZE DESC
;QUIT ;

to calculate the size before, but it does not work with Hadoop.

Any suggestions?

Edit: I had an idea of someone looping the Hadoop library and capturing the data size (unless there is a direct way to do it)

1. Copy a table from Hadoop to Work library

2. Capture the size of that table with name in another table

3. Delete the table from work and copy the second table

4. Do while

This will give me size of all the tables. It is a very lengthy process specially since I have dozens of tables in Hadoop.

Can someone assist me with that code? I do not know Do while loop in SAS EG.

Re: Finding how much space is used in Hadoop using SAS EG

rajdeep — Mon, 12 Aug 2019 12:26:19 GMT

HI Amity,

Please try to run the code. The "hdfs ls" will show the filesize as well in the output list. Like in the code it's in *, but if you need any specific file then you can mention that. Yeah, you need to change the username password and path as per the hadoop location.

The libname what you had mapped in your EG from that you might get the credentials and path probably.

Have a try and check. I hope you are using 9.4.

options set=SAS_HADOOP_CONFIG_PATH="\\sashq\root\u\abcdef\cdh45p1";
options set=SAS_HADOOP_JAR_PATH="\\sashq\root\u\abcdef\cdh45";
options set=SAS_HADOOP_RESTFUL 1;
proc hadoop username='sasabc' password='sasabc' verbose;
   hdfs cat='/user/sasabc/*';
    hdfs ls='/user/sasabc/*';
run;

Re: Finding how much space is used in Hadoop using SAS EG

amity21 — Mon, 12 Aug 2019 14:14:31 GMT

Hello Rajdeep,

Thank you for your help, but the code did not work.

Below is the code I ran

options set=SAS_HADOOP_JAR_PATH="/sas/thirdparty/hadoopp/final";
options set=SAS_HADOOP_CONFIG_PATH="/sas/thirdparty/hadoopp/final";
options set=SAS_HADOOP_RESTFUL 1;

proc hadoop username='SASEG_USERNAME' password='SASEG_PASSWORD' verbose;
   hdfs cat='/user/SASEG_USERNAME/*';
    hdfs ls='/user/SASEG_USERNAME/*';
run;

This generated a bunch of errors.

ERROR: Could not find extension: (libvas-gssapi)
ERROR: Could not find extension: (libvas-gssapi)
ERROR: Could not find extension: (libvas-gssapi)
ERROR: Could not find extension: (/opt/quest/lib64/libvas-gssapi.so)
ERROR: Could not find extension: (/opt/quest/lib64/libvas-gssapi.so)
ERROR: Could not find extension: (/opt/quest/lib64/libvas-gssapi.so)
ERROR: Could not find extension: (libgssapi_krb5)
ERROR: Could not find extension: (libgssapi_krb5)
ERROR: Could not find extension: (libgssapi_krb5)
ERROR: Could not find extension: (libgssapi)
ERROR: Could not find extension: (libgssapi)
ERROR: Could not find extension: (libgssapi)
ERROR: Could not find extension: (libgss)
ERROR: Could not find extension: (libgss)
ERROR: Could not find extension: (libgss)
ERROR: Unable to load extension: (tksecgss)
ERROR: Unable to load the authentication provider tksecgss.
ERROR: Unable to get home directory: HTTP status: Unauthorized (401)
ERROR: Failed URL: http://elxap251.xxxxx.com:50070/webhdfs/v1/?op=GETHOMEDIRECTORY&user.name=SASEG_USERNAME
ERROR: Could not find extension: (libvas-gssapi)
ERROR: Could not find extension: (libvas-gssapi)
ERROR: Could not find extension: (libvas-gssapi)
2                                                          The SAS System                              10:09 Monday, August 12, 2019

ERROR: Could not find extension: (/opt/quest/lib64/libvas-gssapi.so)
ERROR: Could not find extension: (/opt/quest/lib64/libvas-gssapi.so)
ERROR: Could not find extension: (/opt/quest/lib64/libvas-gssapi.so)
ERROR: Could not find extension: (libgssapi_krb5)
ERROR: Could not find extension: (libgssapi_krb5)
ERROR: Could not find extension: (libgssapi_krb5)
ERROR: Could not find extension: (libgssapi)
ERROR: Could not find extension: (libgssapi)
ERROR: Could not find extension: (libgssapi)
ERROR: Could not find extension: (libgss)
ERROR: Could not find extension: (libgss)
ERROR: Could not find extension: (libgss)
ERROR: Unable to load extension: (tksecgss)
ERROR: Unable to load the authentication provider tksecgss.
ERROR: Unable to list directory status: HTTP status: Unauthorized (401)
ERROR: Failed URL: http://elxap251.xxxxx.com:50070/webhdfs/v1/user/SASEG_USERNAME?op=LISTSTATUS&user.name=SASEG_USERNAME
ERROR: Directory /user/SASEG_USERNAME doesn't exist

Re: Finding how much space is used in Hadoop using SAS EG

rajdeep — Mon, 12 Aug 2019 16:37:11 GMT

Hi

Please just confirm me once what version of SAS you are using.

Also, in the area of Hadoop username and password have you passed Hadoop username and password or not!!

Just right click on the library of Hadoop in SAS EG and have a check what username and password is been assigned then try.

Also which path in the same library has been passed just have a check. Don't pass the username and password which you are using to login to SAS EG like for profile credentials.

Re: Finding how much space is used in Hadoop using SAS EG

amity21 — Mon, 12 Aug 2019 16:52:17 GMT

Hi Rajdeep,

Thank you for your response.

SAS EG version is 7.12

I was using SAS EG Username password.

When I right click SAS EG I can't find any login credentials.

Path is

jdbc:hive2://elxap251.xxxxx.com:2181,elxap252.xxxxx.com:2181,elxap253.xxxxx.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2

Only username password I have, beside SAS EG, is Oracle which I do not think applies here either.

Re: Finding how much space is used in Hadoop using SAS EG

rajdeep — Mon, 12 Aug 2019 17:25:08 GMT

Okay. Just hold a sec and please go through the following link once and try to identify where something you are missing out.

https://documentation.sas.com/?docsetId=acreldb&docsetTarget=n1h398otek0j00n1itib5k7ch738.htm&docsetVersion=9.4&locale=en

Also, just like to know when you clicked on the properties of the Hadoop library in SAS EG, you are just getting the path.

Like, I am just talking about the library like SASHELP or SASWORK under app server area.

If you have the access to SAS management console then please do check over there as well the properties of the same library, but before that please go through the link so that you might get some clarity about SAS to Hadoop JDBC bridge.