topic Re: Access to Spark Natively on SAS Viya in SAS Data Science

Access to Spark Natively on SAS Viya

gra_in_aus — Mon, 17 Jun 2024 03:36:52 GMT

Hi,

I am wondering if Spark can be made natively available on SAS Viya? As I don't want to push data outside of the Viya environment.

Many thanks

Re: Access to Spark Natively on SAS Viya

LinusH — Mon, 17 Jun 2024 14:33:50 GMT

Are you asking if you can deploy a Spark engine inside a SAS Viya deployment?

ASFAIK SAS doesn't provide that.

So the question is what is your real constraint?

What data do you have in CAS and how is it stored?

If you chose to store the data in 3rd party DBMS/engine, you can make that available to your Spark cluster of choice.

What options do you have, and where have you deployed Viya?

Re: Access to Spark Natively on SAS Viya

gra_in_aus — Wed, 19 Jun 2024 10:09:46 GMT

My limited understanding of SAS Viya is that it supports open source languages. Hence, as Spark is open source similar to Python, r, Julia, Scala, etc... I was hoping it would be supported without the need to use another platform, i.e. Databricks.

Re: Access to Spark Natively on SAS Viya

LinusH — Thu, 20 Jun 2024 08:37:10 GMT

In my mind Saprk is not a language, it's a platform or engine for data management, query and analysis, similar to the engines Viya provides (CAS and SAS Compute). So it makes little sense to have a Spark cluster within SAS Viya platform. Again, you can deploy open source Spark and have the same data availble to it as you have in SAS.

Spark supports SQL, Python, Java and Scala, (and perhaps more since it's open source), of wichSAS supports more or less SQL, Python and Java.

What is your requirement?

Re: Access to Spark Natively on SAS Viya

gra_in_aus — Tue, 25 Jun 2024 21:18:06 GMT

Requirement is to be able to support developers to develop SAS and upskill on Spark for future Databricks platform enablement.

Re: Access to Spark Natively on SAS Viya

SASKiwi — Wed, 26 Jun 2024 04:29:46 GMT

@gra_in_aus - Here is the documentation link for Viya Spark support. Upskilling on Spark / Databricks would be done on the Spark platform itself. SAS can be used to both query Spark-based data such as Databricks, or to run in-database processes on Spark.

EDIT: Just to be clear Spark / Databricks is a third-party product so it will always sit "outside" of Viya on its own cloud-based platform.