topic Updating data from SAS to HDFS andLASR server in SAS Visual Analytics

Updating data from SAS to HDFS andLASR server

nikhil_khanolkar — Mon, 13 Apr 2015 04:24:55 GMT

Hi,

I have a data in SAS which gets updated at the regular interval (Every Hour). I want to update records from the SAS server in the datasets in HDFS and LASR server.

Any help would be appreciated.

Thanks,

Nikhil

Re: Updating data from SAS to HDFS andLASR server

gergely_batho — Mon, 13 Apr 2015 06:28:36 GMT

Hi,

There are many ways to accomplish this.

By "dataset in HDFS" you mean your data is in SASHDAT format?

What do you mean by "updated"? Do you have new rows, or old rows are changing? Or the whole dataset is completely new?

One example to add (append) observations to a SASHDAT in-memory file:

data HDATLib LASRLib.hdatFile(append=yes);

set sourceLib.sourceFile;

where recordStatus='NEW';/*put a where statement here that identifies new observations. For example: &lastLoadTime. <= timeStamp*/

run;

After that you can reload the table that is in the LASR memory from the SASHDAT table.

After that you can synchronize the SASHDAT table.

But you could update the rows directly in the LASR memory table (using PROC IMSTAT), then synchronize the associated SASHDAT table.

Message was edited by: Gergely Bathó I was mixing up the table options for SASHDAT and SASIOLA. Currently there is NO append=yes table option for the SASHDATengine.

Re: Updating data from SAS to HDFS andLASR server

nikhil_khanolkar — Mon, 13 Apr 2015 06:46:54 GMT

Hi Gergely,

Thanks for Responding.

A) Yes the dataset is in SASHDAT format

B) By update I mean "Old rows are changing"

e.g. Account is the Key variable. We want update the change in Other columns using Key variable Join between SAS and HDFS tables.

Like Proc SQL update.

Nikhil

Re: Updating data from SAS to HDFS andLASR server

gergely_batho — Mon, 13 Apr 2015 07:48:55 GMT

Do you have an associated in-memory table?

First I would update the in-memory table, then synchronize the SASHDAT table (if needed).

The first task can be done with the PROC IMSTAT UPDATE statement:

SAS(R) LASR(TM) Analytic Server 2.2: Reference Guide

The second task with the PROC IMSTAT SAVE stement:

SAS(R) LASR(TM) Analytic Server 2.2: Reference Guide

Re: Updating data from SAS to HDFS andLASR server

gabras — Wed, 04 Jan 2017 15:31:46 GMT

Hi,

i have the same problem but i don't understand how can i solve it.

I have table1 which is in memory and table2 which is the table to use for the update.

table2 can have rows that exist in table1 (update) or new rows (append).

How can i use proc imstat update statement in this case?

Thank you!!!

Re: Updating data from SAS to HDFS andLASR server

gabras — Wed, 04 Jan 2017 17:11:38 GMT

To be more precisely:

data table1;
input ID $ 1-2 data $ 3-10;
ID_2=ID+1;
datalines;
1 sysdate
2 sysdate
3 sysdate
4 sysdate
5 sysdate
6 sysdate
7 sysdate
8 sysdate
9 sysdate
10 sysdate
;
run;

data table2;
input ID $ 1-2 data $ 3-10;
ID_2=ID+20;
datalines;
1 sysdate1
2 sysdate2
3 sysdate3
;
run;

After loading tables in memory (we are in a distributed env) i used this program:

proc imstat data=valibr.table1;
UPDATE data=valibr.table2;
run;
fetch / format ;
quit;

Unfortunately it doesen't work. The result is:

Rows from the table
HPS.TABLE1
ID	data	ID_2
3	sysdate3	23
3	sysdate3	23
3	sysdate3	23
3	sysdate3	23
3	sysdate3	23
3	sysdate3	23
3	sysdate3	23
3	sysdate3	23
3	sysdate3	23
3	sysdate3	23