topic Re: Hadoop: Renaming a HDFS file from SAS in SAS Programming

Hadoop: Renaming a HDFS file from SAS

DrBigAl — Fri, 28 Sep 2018 16:53:21 GMT

Hi, I have successfully loaded data into a hdfs file as a txt/ pipe delimited. My only problem is that the name of the file is not changed.

HDFS path: /home/user/data/

The name of the file is.....

sasdata-2018-09-28-12-32....dlv

I want it to be...

my_file_info.txt

Every time the code is ran, the former name changes, but still keeps the sasdata: prefix. How do I, from SAS, change the name of the .dlv file and rename it to my_file_info.txt, while accounting for the sasdata: name changes?

proc sql;
connect to hadoop (server="&serv" port=&portn. user="&user" password="&dbpass"
SCHEMA=lw subprotocol=hive2 DBMAX_TEXT=&mxtext.);
execute(create external table my_file_info(
num_cd string)

row format delimited fields terminated by '|'
LOCATION "/home/user/data/") by hadoop;
disconnect from hadoop;
quit;

libname lw hadoop server="&serv" port=&portn. user="&user" password="&dbpass"
SCHEMA=lw_research subprotocol=hive2 DBMAX_TEXT=&mxtext.
;

proc sql;
insert into lw.my_file_info select * from &processlib.mytable;
quit;

Thank you for your help!

Re: Hadoop: Renaming a HDFS file from SAS

ChrisNZ — Sun, 30 Sep 2018 21:13:31 GMT

I no longer have access to Hadoop, so my help will be limited.

When is the dlv file created? When you run insert into?

Re: Hadoop: Renaming a HDFS file from SAS

DrBigAl — Mon, 01 Oct 2018 02:41:25 GMT

Yes. It is created when I run the insert to code.

Re: Hadoop: Renaming a HDFS file from SAS

ChrisNZ — Mon, 01 Oct 2018 03:18:35 GMT

Why do you care what the file name is? This is internal Hadoop sausage-making isn't it? You only access the metadata layer (table my_file_info) don't you?

Re: Hadoop: Renaming a HDFS file from SAS

DrBigAl — Mon, 01 Oct 2018 13:03:19 GMT

You are right. A lot of internal sausage-making. If it were up to me, I'd keep it as it is.

Re: Hadoop: Renaming a HDFS file from SAS

ChrisNZ — Mon, 01 Oct 2018 21:22:03 GMT

Sorry, I can't help further. It is a bit too long ago to remember if I went to look at the actual file names, though I suspect I was never interested. Ask tech support if no one here knows.