Access to Spark Natively on SAS Viya

[gra_in_aus]

Hi,

I am wondering if Spark can be made natively available on SAS Viya?  As I don't want to push data outside of the Viya environment.

Many thanks

G

[LinusH]

Are you asking if you can deploy a Spark engine inside a SAS Viya deployment?

ASFAIK SAS doesn't provide that.

So the question is what is your real constraint?

What data do you have in CAS and how is it stored?

If you chose to store the data in 3rd party DBMS/engine, you can make that available to your Spark cluster of choice.

What options do you have, and where have you deployed Viya?

[gra_in_aus]

My limited understanding of SAS Viya is that it supports open source languages. Hence, as Spark is open source similar to Python, r, Julia, Scala, etc... I was hoping it would be supported without the need to use another platform, i.e. Databricks.

[LinusH]

In my mind Saprk is not a language, it's a platform or engine for data management, query and analysis, similar to the engines Viya provides (CAS and SAS Compute). So it makes little sense to have a Spark cluster within SAS Viya platform. Again, you can deploy open source Spark and have the same data availble to it as you have in SAS.

Spark supports SQL, Python, Java and Scala, (and perhaps more since it's open source), of wichSAS supports more or less SQL, Python and Java.

What is your requirement?

[gra_in_aus]

Requirement is to be able to support developers to develop SAS and upskill on Spark for future Databricks platform enablement.

[SASKiwi]

@gra_in_aus - Here is the documentation link for Viya Spark support. Upskilling on Spark / Databricks would be done on the Spark platform itself. SAS can be used to both query Spark-based data such as Databricks, or to run in-database processes on Spark.

EDIT: Just to be clear Spark / Databricks is a third-party product so it will always sit "outside" of Viya on its own cloud-based platform.