Exploring, predicting and reporting with SAS Visual Analytics and SAS Visual Statistics

Updating data from SAS to HDFS andLASR server

Reply
Contributor
Posts: 31

Updating data from SAS to HDFS andLASR server

Hi,

I have a data in SAS which gets updated at the regular interval (Every Hour). I want to update records from the SAS server in the datasets in HDFS and LASR server.

Any help would be appreciated.

Thanks,

Nikhil

SAS Employee
Posts: 340

Re: Updating data from SAS to HDFS andLASR server

Posted in reply to nikhil_khanolkar

Hi,

There are many ways to accomplish this.

By "dataset in HDFS" you mean your data is in SASHDAT format?

What do you mean by "updated"? Do you have new rows, or old rows are changing? Or the whole dataset is completely new?

One example to add (append) observations to a SASHDAT  in-memory file:

data HDATLib  LASRLib.hdatFile(append=yes);

     set sourceLib.sourceFile;

     where recordStatus='NEW';/*put a where statement here that identifies new observations. For example:  &lastLoadTime.  <=  timeStamp*/

run;

After that you can reload the table that is in the LASR memory from the SASHDAT table.

After that you can synchronize the SASHDAT table.


But you could update the rows directly in the LASR memory table (using PROC IMSTAT), then synchronize the associated SASHDAT table.


Message was edited by: Gergely Bathó I was mixing up the table options for SASHDAT and SASIOLA.  Currently  there is NO append=yes table option for the SASHDATengine.

Contributor
Posts: 31

Re: Updating data from SAS to HDFS andLASR server

Posted in reply to gergely_batho

Hi Gergely,

Thanks for Responding.

A) Yes the dataset is in SASHDAT format

B) By update I mean "Old rows are changing"

e.g. Account is the Key variable. We want update the change in Other columns using Key variable Join between SAS and HDFS tables.

       Like Proc SQL update.

Nikhil

SAS Employee
Posts: 340

Re: Updating data from SAS to HDFS andLASR server

Posted in reply to nikhil_khanolkar

Do you have an associated in-memory table?

First I would update the in-memory table, then synchronize the SASHDAT table (if needed).

The first task can be done with the PROC IMSTAT UPDATE statement:

SAS(R) LASR(TM) Analytic Server 2.2: Reference Guide

The second task with the PROC IMSTAT SAVE stement:

SAS(R) LASR(TM) Analytic Server 2.2: Reference Guide

Contributor
Posts: 20

Re: Updating data from SAS to HDFS andLASR server

Posted in reply to gergely_batho

Hi, 

i have the same problem but i don't understand how can i solve it.

 

I have table1 which is in memory and table2 which is the table to use for the update. 

table2 can have rows that exist in table1 (update) or new rows (append).

 

How can i use proc imstat update statement in this case?

 

Thank you!!!

Contributor
Posts: 20

Re: Updating data from SAS to HDFS andLASR server

To be more precisely:


data table1;
input ID $ 1-2 data $ 3-10;
ID_2=ID+1;
datalines;
1 sysdate
2 sysdate
3 sysdate
4 sysdate
5 sysdate
6 sysdate
7 sysdate
8 sysdate
9 sysdate
10 sysdate
;
run;

 


data table2;
input ID $ 1-2 data $ 3-10;
ID_2=ID+20;
datalines;
1 sysdate1
2 sysdate2
3 sysdate3
;
run;

 

After loading tables in memory (we are in a distributed env) i used this program:


proc imstat data=valibr.table1;
UPDATE data=valibr.table2;
run;
fetch / format ;
quit;

 

Unfortunately it doesen't work. The result is:

Rows from the table
HPS.TABLE1
IDdataID_2
3sysdate323
3sysdate323
3sysdate323
3sysdate323
3sysdate323
3sysdate323
3sysdate323
3sysdate323
3sysdate323
3sysdate323

 

Ask a Question
Discussion stats
  • 5 replies
  • 727 views
  • 0 likes
  • 3 in conversation