topic Re: Reloading in-memory table when sashdat file has been replaced in SAS Viya

Reloading in-memory table when sashdat file has been replaced

rgreen33 — Thu, 04 Nov 2021 20:09:24 GMT

I have a folder setup on my SAS Viya server, where sashdat files are being dropped. I am working on a SAS Viya Studio script that will do 2 things...

If a new file is found, which has never been loaded into memory, the script will load the file into memory and promote it so that it is available to SAS Viya Visual Analytics.
1. I have this part working, using "proc cas".
If a sashdat is updated (replaced by a new version), I want to drop and reload the in-memory data.
1. This is the part that I am struggling with. I was thinking that I could compare the date/time on the file with the loaded (in-memory) date/time, but I have not yet found a way to do this.

Any suggestions for part 2 above?

Thanks,

Ricky

Re: Reloading in-memory table when sashdat file has been replaced

gwootton — Thu, 04 Nov 2021 20:24:09 GMT

Do the included sample jobs for CAS table management not work for this?

Sample Jobs in SAS Environment Manager
https://go.documentation.sas.com/doc/en/calcdc/3.5/caldatamgmtcas/n150v9svjp42ron1hn9r496xj49m.htm#p1lsy0yhdi5q5ln1fapbsmsuco6d

I think the load job will reload any that have updated data in their source.

Re: Reloading in-memory table when sashdat file has been replaced

rgreen33 — Fri, 05 Nov 2021 11:40:50 GMT

Perfect! I did not realize that the refresh would work like this. Thank you so much for the link. I was making this harder than it needed to be.

I'm now rethinking the script that I already have written. From the above link, if I setup a "refresh" job, any ideas on how it would work if a new sashdat file is encountered (one where the table is not yet in memory)? Since the "target" does not exist, will it skip it, or will it go ahead and load it?

I'm also now wondering if the sashdat file will be able to be replaced for those files that have already been loaded into memory??? I'm wondering if I should allow users to upload files to a different directory...and then import them into my CASLIB (folder). I'm trying to understand what the "import" job would do if I set it to newer...to import ONLY the files that have been updated.

Ideas on how to piece this together to make it work?

Thanks,
Ricky

Re: Reloading in-memory table when sashdat file has been replaced

gwootton — Fri, 05 Nov 2021 13:25:38 GMT

I believe by default the load job will load any sashdat files it finds in the path, similarly import when set to newer I think would capture new objects as "newer" than nothing. You can always give it a try and see what works for you.

Re: Reloading in-memory table when sashdat file has been replaced

elisehoo1 — Fri, 05 Nov 2021 13:52:55 GMT

Hi Ricky ,

any luck ? may i know how do you solve it to have a "autorefresh" from sashdat if the in memory table does not avalable?

cheers

Re: Reloading in-memory table when sashdat file has been replaced

rgreen33 — Fri, 05 Nov 2021 17:41:17 GMT

@elisehoo1 ,

I am still testing things. I will post the solution once I have things working.

Thanks,

Ricky