VA LASR back up

[DNAFoundry]

Hi,

If i upload data into LASR using VA, does that have persistance in case of a system reset?

If i want to manage my tables in VA, what should i have in place to ensure it is reloaded into memory whenever required and how should i manage backups?

[JuanS_OCS]

Hello @DNAFoundry,

since LASR tables are in-memory tables, the only ways to keep them on the disk, as far as I am aware of, are:

• If your VA server is working on a virtual machine, when you get a snapshot (a "server backup"), you can specify you would like to save also the state of the RAM memory. Therefore, if you have a reboot, or when you need to restore the server back for disaster recovery, you can load the disks and the memory, altogether. Only downside, is that in this case, the snapshot will take longer and will be large on storage.
• The other option is to save the LASR tables into HDFS, which is available only if you have a distributed version of SAS Visual Analytics, meaning distributed on several nodes and with a Hadoop cluster.

For the second option, you can give a look to this post: https://communities.sas.com/t5/SAS-Visual-Analytics/Copying-data-into-HDFS-from-LASR-server/td-p/196...

or to transform this manual action: http://support.sas.com/documentation/cdl/en/vaug/67500/HTML/default/viewer.htm#n16p1wynamhmasn1otljh...

into code: http://support.sas.com/documentation/cdl/en/inmsref/67213/HTML/default/viewer.htm#n1qjni2w84yi8nn13v...

https://communities.sas.com/t5/SAS-Visual-Analytics/Copying-data-into-HDFS-from-LASR-server/td-p/196...

[JuanS_OCS]

Maybe this technical paper would help you as well ( Techniques in Processing Data on Hadoop 😞 https://support.sas.com/resources/papers/proceedings14/SAS033-2014.pdf

[DNAFoundry]

Hi,

I am not talking about keeping it on disk. I mean what is the accepted procedure for working with in memory tables:

If i have uploaded 25,000 different csv files to the database and I need to reset the system. Is my only option to find and reload all files? Or have hadoop (which again the SAS team couldnt figure out how to install)?

If I have LASR and suffer a power outage, what exactly am i supposed to do? Im just trying ot get my head around how this is being used inb production by large companies - are they really snapshotting the virtual machines? 

[JuanS_OCS]

It depends on the approach. But it seems you will take the most simple one: I understand you will create your own SAS tables or csv files out of SAS, and then let VA to load them into memory.

In case you just work with csv files and/or SAS tables, you already have the tables on the disk, and then you probably want to use the Autoload functionality. You drop the tables on a certain folder and the system itself will autload them automatically (or update them, if already loaded). Simplifying what is already simple.

http://blogs.sas.com/content/sgf/2016/04/04/sas-visual-analytics-autoload-configuration-made-easy/

http://support.sas.com/documentation/onlinedoc/va/7.3/en/vaag.pdf

The Hadoop approach is interesting only if you have VA on different servers (distributed) and if you run queries on the VA Data Builder that generate data that is only in LASR, not on the disk.

[DNAFoundry]

But if I am uploading csv files using VA, how can I upload files into folders on the server?

Are these two options honestly how a bank or large company do this? Potentiall millions of files on a disk somewhere?

[JuanS_OCS]

You can. Please read http://support.sas.com/documentation/onlinedoc/va/7.3/en/vaag.pdf on page 22, the Autoload chapter.

To keep it simple:

- You have already a default Autoload folder:

Under Config/Lev1/AppData/SASVisualAnalytics/VisualAnalyticsAdministrator, you will find a folder named Autoload.

You can drop your files to be autoloaded in this folder and it will autolad them on the best way it can, so ensure they are on the right format, comma delimited, etc. Or just drop SAS tables in this folder.

- You have the scripts to schedule (cron if you are on Windows, or Windows Task Scheduler, if you are on a Windows server) at Config/Lev1/Applications/SASVisualAnalytics/VisualAnalyticsAdministrator,

This folder contains a schedule.bat or schedule.sh. You must run this script with a user that is registered on the SAS metadata and has the right permissions on the relevant metadata folders/items and the 2 operating system folders mentioned above. This script will schedule the task to run every 15 min by default, but it can be changed.

Besides this default autoload folder, you can create more of your own (say, for different departments), by following these instructions: http://blogs.sas.com/content/sgf/2016/04/04/sas-visual-analytics-autoload-configuration-made-easy/ which are also in the Administration guide from the first link in this post

[DNAFoundry]

My uplodad folder is on a linuix server that I have to ftp into. The instrucitons seem to show a windows based directory on a machine that someone is using.

When I upload a csv file in VA, does it copy the file into this folder and manage all the autoloading? Or do I have to configure this functionality somewhere?

It seems like unless the file exists on the server the autoload does not work. If I only upload files in VA then if the filess are not copied over (as well as loaded into in memory storage) then I cant administer the system only through VA.

[JuanS_OCS]

Indeed, the file must be on the folder at the VA server.

If you server is Linux, you can ask your administrators to create a samba share (very easy, it should consume not much more than 5 or 10 min), that you can reach from any other Linux or Windows server or pc, with the right credentials and shared folder, that the Linux administrators should provide to you.

Or they can create a symbolic link from an ftp folder to the SAS Autoload folder, so when you move data to the ftp folder, it is actually put on the VA Autoload folder.

Either option, please ensure the folder structure behind Autoload, and the permissions are kept.

About your other question, if you load the file manually, it won;t be moved to the autoload folder and you will need to reload the data manually every time.

[DNAFoundry]

Ok, well I guess that means I can't use VA to administer the system by loading in CSV files as there is no route directly from VA to having the data be recovered on event of in memory storage being lost.

[JuanS_OCS]

The message is exactly the opposite so: which part of the message was not clear?

[DNAFoundry]

I want to use VA web interface to upload files into tables on LASR in a way that is persistant. If i upload via the web interface then the file is lost and the data only exists in memory. And as the file is lost the data can't be autoloaded.

The resources you sent mean that I have to load files into LASR by copying them accross to a folder, not via the VA web interface. This is a completly different system and workflow. Am I missing something?

[JuanS_OCS]

Not necessarily to copy them (no requirement of an extra action), but your admin will need to make the csv files locally available to the VA server, at least through a shared folder. Then you can have Autoload, DataBuilder Queries, Interactive Load, ReloadonStart.... many options. I still think that Autoload is the most simple and easier option, but the choice is yours.

If you have the opportunity to read "About Loading Data" from the Administration Guide, pages 100-11-13, or in general the chapter 2, it should cover quite well all your questions.

http://support.sas.com/documentation/onlinedoc/va/7.3/en/vaag.pdf

[JuanS_OCS]

[DNAFoundry]

If you want to have any persistance you have to have an admin move csv files onto the server. That is different to uploading data simply via the web interface - which is what I was wanting to do.