topic Re: Must Reload-on-Start backup storage be loaded manually? in SAS Visual Analytics

Must Reload-on-Start backup storage be loaded manually?

RogerHed — Wed, 15 May 2019 14:03:36 GMT

Reading the Administrator instructions and activating the VA Reload-on-Start we interpreted that SAS automatically sends backup data when loading data into LASR memory. Seems not to be correct. Just to get it right, do we have to load backup data first and then send this to LASR memory?

Re: Must Reload-on-Start backup storage be loaded manually?

alexal — Wed, 15 May 2019 14:24:46 GMT

@RogerHed ,

Just to get it right, do we have to load backup data first and then send this to LASR memory?

That called the Autoload process.

Re: Must Reload-on-Start backup storage be loaded manually?

RogerHed — Wed, 15 May 2019 14:31:05 GMT

No, we don't use the Autoload. We load data directly to VALIBLA. Applying the Reload-on-start did however not replicated the load to the backup path. The question is if this is supposed to be done by the Reload-on-start or manually?

Re: Must Reload-on-Start backup storage be loaded manually?

alexal — Wed, 15 May 2019 14:34:52 GMT

@RogerHed ,

SAS automatically places a data set copy of the source data in the data provider library that is the designated backing store for the target LASR library. You do not need to do anything manually, just enable Reload-On-Start and load the data to LASR.

Re: Must Reload-on-Start backup storage be loaded manually?

RogerHed — Wed, 15 May 2019 15:01:35 GMT

Great. So this means we have something wrong since the data provider library is not updated on load LASR.
Was curious about the authorization since this is not mentioned in the Admin guide instructions. In Metadata we have same settings on VALIBLA as on the data provider. Running on a Linux. The physical authorization group on the folder may not work. We did a test with a user with physical and Metadata rights to load VALIBLA and Backup directory. But this did not create the backup table anyway.
So anything else we need to consider?

Re: Must Reload-on-Start backup storage be loaded manually?

Madelyn_SAS — Wed, 15 May 2019 15:22:17 GMT

How are you loading the tables? The tables that are eligible to participate in reload-on-start are:

Participating tables from imports of local files.

Participating tables from imports of Google Analytics, Facebook, or Twitter data.

Additional considerations are here: Additional Considerations

You could also check the log for reload-on-start. It will be in a location similar to this:

/Applications/SASVisualAnalytics/VisualAnalyticsAdministrator/Monitoring/Logs

Re: Must Reload-on-Start backup storage be loaded manually?

RogerHed — Thu, 16 May 2019 08:39:04 GMT

Let me clarify. Reload do function as expected when starting up. It's the backup replication that reload doesn't do. As Alexal writes "SAS automatically places a data set copy of the source data in the data provider library", this never happens. Nor is this action logged in any log (perhaps due to suppressed log configuration).

Have checked the .../Monitoring/Logs.

Re: Must Reload-on-Start backup storage be loaded manually?

alexal — Thu, 16 May 2019 11:20:53 GMT

@RogerHed ,

this never happens.

I would like to review a SAS log from the loading attempt.

Re: Must Reload-on-Start backup storage be loaded manually?

RogerHed — Thu, 16 May 2019 12:45:20 GMT

Attached the log from a manual upload made from VA Administrator (following the example given in How Reload-on-Start Works).

Before this we checked the permissions was correct for the user testing: LASR Server=RM, Library=RM, Folder=RM, Table=RM, R, WM, W and Linux RWX on Group. We also tried to register the backup table before upload if that whould be an issue.

The user that started the LASR do have:

The identity that starts the server has metadata-layer access to the table, its parent folder, and its parent library.
The identity that starts the server has host access to the table (in the associated data provider library).

We are stuck!

Re: Must Reload-on-Start backup storage be loaded manually?

alexal — Thu, 16 May 2019 13:09:33 GMT

@RogerHed ,

Thanks for your response. Here is what is going on. You are trying to load the data from the HPS library:

proc lasr port=10011
data=HPS.Email_recipients ( label='Inläst torsdagen den 16:e maj 2019 kl. 14:10:48 GMT+0200 från "control.Control.Email_recipients" av "XXX"')
signer="http://XXX:7980/SASLASRAuthorization"
add noclass;
performance
host="XXX"
;
run;

Which was defined as a Base SAS library, located on the server:

NOTE: Libref HPS was successfully assigned as follows:
Engine: V9
Physical Name: /sas/xxx/data/control

As @Madelyn_SAS said before, The tables that are eligible to participate in reload-on-start are:

Participating tables from imports of local files.
Participating tables from imports of Google Analytics, Facebook, or Twitter data.

Here is a piece of SAS code from a loading attempt of the local file. You will see there will be another library called DPPUBLIC. A local file will be imported to that library first (if Reload-On-Start was enabled):

2019-05-16T08:56:39,816 INFO [00000012] :sasdemo - 17 Options VALIDVARNAME=ANY VALIDMEMNAME=EXTEND NO$SYNTAXCHECK;
2019-05-16T08:56:39,817 INFO [00000012] :sasdemo - 18 LIBNAME DPPUBLIC BASE
2019-05-16T08:56:39,817 INFO [00000012] :sasdemo - 18 ! "/<SASConfig/LevX/AppData/SASVisualAnalytics/VisualAnalyticsAdministrator/PublicDataProvider";
2019-05-16T08:56:39,817 INFO [00000012] :sasdemo - NOTE: Libref DPPUBLIC was successfully assigned as follows:
2019-05-16T08:56:39,817 INFO [00000012] :sasdemo - Engine: BASE
2019-05-16T08:56:39,817 INFO [00000012] :sasdemo - Physical Name: /<SASConfig/LevX/AppData/SASVisualAnalytics/VisualAnalyticsAdministrator/PublicDataProvider
2019-05-16T08:56:39,817 INFO [00000012] :sasdemo - 19
2019-05-16T08:56:39,819 INFO [00000012] :sasdemo - 4 The SAS System Thursday, May 16, 2019 08:56:00 AM
2019-05-16T08:56:39,819 INFO [00000012] :sasdemo -
2019-05-16T08:56:39,819 INFO [00000012] :sasdemo - 20 Options VALIDVARNAME=ANY VALIDMEMNAME=EXTEND NO$SYNTAXCHECK;
2019-05-16T08:56:39,819 INFO [00000012] :sasdemo - 21 %let EFI_QUOTED_NUMERICS = yes;
2019-05-16T08:56:39,819 INFO [00000012] :sasdemo - 22 %let EFI_MISSING_NUMERICS = yes;
2019-05-16T08:56:39,819 INFO [00000012] :sasdemo - 23 FILENAME extfile
2019-05-16T08:56:39,819 INFO [00000012] :sasdemo - 23 ! "/tmp/SAS_work0AC6000077DE_lasr.sas.com/SAS_workB32E000077DE_lasr.sas.com/50E1BA44D3DAD124ED98870B9A6E21D
2019-05-16T08:56:39,819 INFO [00000012] :sasdemo - 23 ! D.82ba30d1d01eebde7fb388d12bb52ece_SASServer12_10.csv" ENCODING="UTF-8";
2019-05-16T08:56:39,832 INFO [00000015] :sasdemo - 24 PROC IMPORT DATAFILE=extfile OUT=DPPUBLIC.dp_test
2019-05-16T08:56:39,833 INFO [00000015] :sasdemo - 25 DBMS=CSV REPLACE;
2019-05-16T08:56:39,834 INFO [00000015] :sasdemo - 25 ! DELIMITER=','; DATAROW=2; GUESSINGROWS=502; RUN;

Then from DPPUBLIC to the LASR server:

2019-05-16T08:56:39,965 INFO [00000012] :sasdemo - 78 Options VALIDVARNAME=ANY VALIDMEMNAME=EXTEND NO$SYNTAXCHECK;
2019-05-16T08:56:39,967 INFO [00000020] :sasdemo - 79 DATA LASRLIB.dp_test(LABEL='Imported on Thursday, May 16, 2019 08:56:27 AM GMT-0400 from "dp_test.csv" by "sasdemo"');
2019-05-16T08:56:39,967 INFO [00000020] :sasdemo - 79 ! SET DPPUBLIC.dp_test; RUN;

I hope that makes sense. Let me know if you have any questions.

Re: Must Reload-on-Start backup storage be loaded manually?

RogerHed — Fri, 17 May 2019 06:01:36 GMT

Thank you Alexal!
We are uploading from Library reference CONTROL that is a BASE library. It seems this is reassigned by the code to "libname HPS 'control' loglib=yes;". I know the HPS was tagged in the data provider library. Will check if this is correctly done.

Re: Must Reload-on-Start backup storage be loaded manually?

RogerHed — Fri, 17 May 2019 09:18:30 GMT

Since upload from VA Administrator seems to use a strange method assigning HPS to the SAS BASE library, we created a simple test job in DI Studio. This is our normal way to load data into LASR (VALIBLA) and see Reload-on-Start doesn't create or update the Data Provider (backup).

Providing our settings and the test run from DI Studio Job. As I see it it's very clear data comes from a local SAS BASE Table and not from Metadata My Folder location. The user that started the LASR do have metadata-layer access to the table. The user that starts the server has host access to the table in the associated data provider library (it's new and don't exist in the data provider location yet, both user starting LASR and user running job has write access to local folder).

Do anyone get anything out of this?

Re: Must Reload-on-Start backup storage be loaded manually?

alexal — Fri, 17 May 2019 10:34:26 GMT

@RogerHed ,

My feedback to that will be the same - you are using the wrong loading mechanism. Please open Data Explorer and try to load local data:

Re: Must Reload-on-Start backup storage be loaded manually?

RogerHed — Fri, 17 May 2019 12:05:23 GMT

OK, we tested this approach and it works when true Local Table. We also tested same Importer but with Server Table and it imports but don't create the backup table. Conclusion is that Reload is a general function but the creation of the Reload-data (Data provider copy) is strictly limited to user uploads.

Will put this reflection to our SAS Support and ask reason for not use the Data Provider on other direct loads (as from Batch). Since we use the direct-load-approach in DI Studio instead of Autoload, we need the Batch code to replicate to the Data provider in order to reload correct data version on server or service restarts.

Re: Must Reload-on-Start backup storage be loaded manually?

alexal — Fri, 17 May 2019 17:47:58 GMT

@RogerHed ,

. Conclusion is that Reload is a general function but the creation of the Reload-data (Data provider copy) is strictly limited to user uploads.

Yes, that is correct. Like we mentioned a few times before, not all tables can participate in reload-on-start. Please read the following page: Reload Methods.

Re: Must Reload-on-Start backup storage be loaded manually?

RogerHed — Mon, 20 May 2019 06:08:39 GMT

Not completely true. Reload-on-Start has two parts 1) Load the Data Provider, 2) Reload from Data Provider when starting LASR. The second works for all tables existing on the Data Provider. The "participating" isn't that binary considering this, it's just the first part that is limited.

Since this don't make complete sense we now have asked SAS Support if this is correct and advice using DI Studio Transform that loads LASR directly. The answer is obviously currently to manually load the Data Provider from the DI Jobs. Just making a point that the Transform could be doing this as an Option "Load Reload-on-Start Data Provider Library" or even automatically when Reload is activated in Metadata.

Thank you all for your help in understanding this. I will update with any informational answers from the Support.

Re: Must Reload-on-Start backup storage be loaded manually?

alexal — Mon, 20 May 2019 12:15:53 GMT

@RogerHed ,

The second works for all tables existing on the Data Provider.

That is true, but only for those tables that were registered in the metadata. Your tables weren't registered in the metadata. As I said in my first message, the autoload process suits better for the task you are trying to do.

Since this don't make complete sense we now have asked SAS Support

I work in SAS Technical Support.