VA to manage system

[DNAFoundry]

I am looking at the SAS packages that I have inherited which are SAS DI studio, SAS enterprise guide and visual analytics.

Apart from being in flash (wtf seriously?) I really can't see any reason to not manage the whole system in VA.

As far as I can see I can upload into tables, create views, create overnight jobs that do calculations. Any ETL I can do offline.

Am I missing something? What is the point of DI studio and enterprise guide?

Does anyone manage their whole system in VA? If so, what should I watch out for.

[LinusH]

Enterprise Guide should not mainly be used as a data admin/batch tool. Is rather for data analysts/scientists etc.

For DI Studio - is mainly for the maintenance part of the batch programs, using design GUI, check-in - check-out etc capabilities.

There is a poor mans ETL UI in VA (Data Builder), and you could perhaps stick to that if you have are certain that you can solve ETL logic with mainly SQL driven logic.

So, it depends on the nature of your environment, the more complex, the more need for DI Studio.

And depending on the nature of your analysis requirements, the more complex, the more need for EG.

[DNAFoundry]

I have to do ETL offline using my own software due to the immense differences in the data coming from various instruments we use.

I think complexity is way down the line on this project - it seems basic funcitonality is hard enough to get done for me. I just need to put a whole bunch of number somewhere for now.

Is data being loaded into LASR from VA essentially no different from that loaded from DI?

[JuanS_OCS · Accepted solution]

Hello @DNAFoundry,

your question/s are really legit and good. I also can see some frustration, and I would like to help if possible, from my experience. Let me go through them, maybe we can make sense out of it.

• Flash: you probably would like to upgrade your VA server to 7.4, or to install the brand new SAS VIsual Analytics 8.1. There are huge improvements regarding using more HTML5 and less Flash (no flash at all in 8.1).
• ETLs: 
  • In one side, at this moment, the VA Data Builder is not the best ETL environment... mainly because it is not intended for it! In other hand, while you have some tools (as VA Data Builder), VA is not intended "as is" to create or execute any ETL. This means: VA intention is only to provide High Performance on Analysis of the data (Explorations) and reporting data (Reports). Therefore it expects you to just load the data into LASR and do not query or transforms the data (and even on reports, they sometimes need some optimization). So the data must be transformed before getting into VA. To VA you should just drop plain data to be uploaded directly to LASR.
  • Secondly, and related to the first point, this also means that any query or transformation, you better do it out from VA, another server. Once the transformation is finished, the final tables you can load to VA-LASR.
  • And third, license and legally-wise, you are entitled to perform certain actions on a VA server, depending on your contract with SAS. You probably would like to review this contract or with a SAS representative. Normally this item was discussed during the sales stage.
  • And here is where DI and EG come to the game:
    • EG is a tool that is intended to be used by both experienced and unexperienced users, to query the data with a good GUI. Many people used (and keep using) EG for ETLs, although DI is way better for managing ETLs, mainly because of the code reusability and maintenance. Now it is a bit inproved thanks to the integration with GitHub, for version control.
      • Also, EG is mostly used on VA environments in order to control the load of the data from a client or a server with SAS Base, into the LASR server. In this case, you would need the extra license of the LASR Tools and/or SAS/Connect.
    • DI is at this moment the key tool for managing ETLs. And the reason is quite basic: you can connect to the metadata, and store every step, job, transformation, in the metadata itself, it can be modular and you can reuse easily every piece of it, by adding a simple version control with Check in and check out system.
    • Finally, and although not mentioned by you, I would like to add SAS Studio as 3rd tool, good to mention. Reasons: SAS Studio is the perfect match for SAS Base users, I see it will integrate more and more EG functionality (perhaps also DI in the future, we shall see), and at this moment this product is always included when you have SAS Base and SAS Metadata server in your license. Did I say I love SAS Studio?

All in all, and after this kind-of-tour over some SAS clients, my overview:

Technically, and answering your direct question: you can manage everything within VA itself. SAS Base is there, technically you don't need metadata for transformations of data to be available for VA. But the "buts" come to the game, and it is up to you to consider the pros and cons:

• The bounding legal aspect of your contract is an important actor. Always good to review. Your license of SAS Base probably it is restricted for pure VA actions.
• Even if your contract allows you, high performance happens on VA-LASR basically because SASWORK is almost not in use, no IO usage, almost, because basically it only loads data into RAM memory, and everything afterwards should happen in memory. You can overrule this fact, but then you will lose performance and the mains of VA.
• In order for this to happen, your contract/license should include license to install at least SAS Base and Integration technologies on a different box. So you can run ETLs on the separated box and just copy/load data into VA. And of course your technical team would need to do this installation/deployment.

I hope this long post would solve some of the questions. Please let us know if you have further questions, concerns or remarks.

Best,

Juan

[DNAFoundry]

Thanks for the help.

I think for the timebeing all i need is VA, I am doing all ETL outside and just want to directly load into LASR.

Really just trying to get very basic functionality built after the SAS team couldn't deliver.

[JuanS_OCS]

I think it makes completely sense. I always subscribe to the options to start small and simple, and to start to grow from there.

Many thanks for accepting the post as solution, glad that this quite long message helped to clean the air a bit.

Good luck!