topic Re: Databricks vs SAS in SAS Programming

Databricks vs SAS

jimbobob — Tue, 27 Jun 2023 14:49:29 GMT

Morning SAS Programmers, last night our new head of IT wants to drop SAS from our tool belt. We currently have SAS Viya 3.5 with the AI/ML and Visual Analytics. I come from 25 years of using various versions of SAS from 8 to 9.4 and now mostly use SAS studio. Love the SAS language and all the capabilities I've seen from SAS throughout the years, every bank I've worked for I'm an advocate for bringing in SAS, and this bank is no different. I brought SAS in and have one year left on our contract. My new boss shared an article: Introduction to Databricks and PySpark for SAS Developers | Databricks Blog I would love the communities feedback, especially if anyone has used Pyspark/Databricks and how they think it really compares to SAS's capabilities. How accurate is this article, any feedback on how to counter this article. Please share any articles you find. As always really appreciate this communities thoughts. Thank you.

Re: Databricks vs SAS

Quentin — Tue, 27 Jun 2023 16:07:59 GMT

I also would be interested in thoughts and experiences of comparative evaluations between Databricks and SAS. Databricks seems to be getting a lot of buzz recently, and that will likely only grow with their recent AI acquisition https://techcrunch.com/2023/06/26/databricks-picks-up-mosaicml-an-openai-competitor-for-1-3b/ .

Re: Databricks vs SAS

ballardw — Tue, 27 Jun 2023 16:08:53 GMT

A reply based only on reading that article as I have no experience with the products.

Near the top of the article is a statement: "The PySpark DataFrame API has most of those same capabilities." and shows a trivial data step and apparently analogous code for PySpark. I would determine which capabilities SAS has that this brief example does not match. Then see if any of those are critical to your current processes. Again, I have no experience with that product but if you have use a lot SAS Merge, Update or Modify data steps (just as an example) and those happen to be not one of those "same capabilities" or require lots of code to migrate then those points would be worth addressing.

The example addressing SQL shows an example done in SAS Proc SQL that would execute much faster as Proc Summary code. I would for other examples where use and comparison of SQL is proposed in lieu of other approaches that use other procedures that may have similar performance impact (assumes you actually use Summary instead of SQL for similar simple sums).

The Venn diagram of features seems to pretty much ignore all of the SAS products outside of Foundation and Stat (if I understand the claim) So perhaps bring up the modules you may use not addressed in that diagram.

There is a bit addressing "Column Oriented vs Business Logic" that compares some SAS IF/Then/Else statements with the PsSpark equivalent(? anytime I see code with multiple """ in a single statement I cringe). I suspect whoever prepared that may not be familiar with the SAS Select/When but that may be trivial.

There are sections about what SAS does "better" but I think misses a few points in the bit about Formats: Not common but when helpful the MULTILABEL format supported by summary/means, report and tabulate can move a lot of logic out of other places. Couple that with the PRELOADFMT to report on things that don't appear in a specific analysis sets intermittently can be quit powerful reporting tools. I didn't see anything about data-set driven creation of formats/informats either.

Re: Databricks vs SAS

PaigeMiller — Tue, 27 Jun 2023 19:36:21 GMT

Please also remember that most of these times when a company wants to stop using SAS, it is almost always driven by cost. Whether the article is accurate or not is pretty much irrelevant, in the minds of some managers, the low cost solution wins!

Re: Databricks vs SAS

Reeza — Tue, 27 Jun 2023 19:52:07 GMT

Do you have all programmers or people who use the GUI, EG/Studio Task interface? Databricks handles the AI/ML part well. It also does delta lakes very well and datawarehouseing, which is not something SAS typically does. I don't think Visual Analytics and Databricks Visualizations are on the same level but I haven't used Databricks extensively, we're just implementing it now. The part that Databricks is missing though is that self serve data exploration and manipulation without code.

Re: Databricks vs SAS

SASKiwi — Tue, 27 Jun 2023 20:23:16 GMT

I'm with @PaigeMiller on this one. Any time any company or management talk about replacing SAS with some other product it is all about the software cost and not about the capabilities. That article you shared has a very narrow focus and just really covers a few technical features. I don't regard it as a basis for deciding to choose Databricks over SAS.

However if you are already a longtime SAS site, how many SAS applications do you have and what would be the cost of converting these to another product? This is where many managers become short-sighted. The software costs are peanuts compared with the cost of software conversion. In the case of the SAS applications I look after, the dollar cost of conversion would be into 8 digits, many times the software licensing costs. And that's quite apart from the challenge of resourcing it...

Re: Databricks vs SAS

Quentin — Tue, 27 Jun 2023 20:48:33 GMT

I don't know that companies are considering replacing SAS purely for cost reasons.

I am not an AI/ML person. But my understanding from some of the AI/ML people in my organization is that new models/algorithms are often available in python long before they would be available in SAS, and this is a reason they are happy with Databricks (PySpark). (And yes, I know that SAS and Viya are Python-friendly.) Ten years ago, there may have been some naive people thinking "lets replace PC SAS with python/R because they're free", but I don't think that's necessarily the case today. Databricks is not cheap.

On a more basic level, I knew a group that developed DI studio ETL jobs in SAS 9. They used DI studio jobs, not code. When they attempted to migrate to Viya (I can't remember which version), they discovered that many of the point-and-click DI Studio transformations they used were not available yet in Viya, and that they would be migrated as user-written code rather than transformation objects. This led them to evaluate other non-code ETL products rather than move on with Viya.

I'm a happy SAS user/advocate/etc. So I'm definitely not arguing for Databricks. But as an advocate for SAS, I don't think we should assume that when companies consider replacing SAS, they are driven only by mistaken cost assumptions.

I'm curious, @Reeza , since your company is implementing Databricks now. Do you think the company sees Datababricks as providing value along side Viya, or is it seen as an alternative to Viya?

Re: Databricks vs SAS

PaigeMiller — Tue, 27 Jun 2023 21:03:36 GMT

@Quentin wrote:

I don't know that companies are considering replacing SAS purely for cost reasons.

It happened to me on a previous job. I left rather than start over learning some other programming language.

Re: Databricks vs SAS

Reeza — Tue, 27 Jun 2023 21:49:22 GMT

Sadly haven't worked in a SAS shop for about 4 years now. Cost isn't the issue.

It's about 10x harder to find a trained SAS person than it is to find a R/Python programmer these days though and that's a bigger issue for us. Newer algorithms aren't as important in our world, but the feature set in Python/R or the ability to find easier answers on SO or assistance does matter. Databricks also bills as used, so companies like that feature as well. I prefer dataiku personally so trying to push to that platform instead.

Re: Databricks vs SAS

SASKiwi — Wed, 28 Jun 2023 01:52:29 GMT

FYI SAS Viya has a "bill by user numbers" charging model too, different to SAS 9.4's by product and number of cores charging.

The trend these days seems to be companies only hiring experienced developers, and not training in new software. So if you didn't learn it at university or some other means then tough...not just a SAS issue really.

I've seen Databrick's marketing strategy firsthand - touting themselves as the "SAS replacement". Like any marketing strategy, use your own judgement.

Re: Databricks vs SAS

mkeintz — Wed, 28 Jun 2023 02:26:28 GMT

@Reeza wrote:
Sadly haven't worked in a SAS shop for about 4 years now. Cost isn't the issue.

It's about 10x harder to find a trained SAS person than it is to find a R/Python programmer these days though and that's a bigger issue for us.
...

In a way, this is a cost issue - the indirect cost of finding the SAS programmers.

Re: Databricks vs SAS

Quentin — Wed, 28 Jun 2023 11:41:48 GMT

Sorry to hear your not in a SAS shop anymore, @Reeza . Glad you're still active in the community!

I hadn't thought about the marketplace for hiring candidates, but that makes sense as a business issue.

Re: Databricks vs SAS

PaigeMiller — Wed, 28 Jun 2023 12:41:58 GMT

@mkeintz wrote:

@Reeza wrote:
Sadly haven't worked in a SAS shop for about 4 years now. Cost isn't the issue.

It's about 10x harder to find a trained SAS person than it is to find a R/Python programmer these days though and that's a bigger issue for us.
...

In a way, this is a cost issue - the indirect cost of finding the SAS programmers.

In my case, when the small company I worked for was purchased by a larger company, I was the only SAS programmer they needed and we had one SAS license. So finding a SAS programmer wasn't really an issue. It was simply that the $X thousand dollars we were paying to SAS each year could be replaced by paying zero dollars for R. The fact that I would have had to spend 6-12 months learning R and then writing code to replace the existing SAS code, while at the same time doing nothing else valuable, didn't seem to make a difference to the new company, they were cutting costs! Of course, every company and every person has a different story.

Re: Databricks vs SAS

Reeza — Wed, 28 Jun 2023 15:20:52 GMT

In our company we run Posit/RStudio Server so there are still R costs for us. Things like package management are important when you have bigger teams and production jobs.

Re: Databricks vs SAS

Quentin — Wed, 28 Jun 2023 16:03:27 GMT

It's been interesting to see Rstudio pivot into Posit and embrace Python.

So now you've got Posit with an environment for R and Python. And Anaconda as an environment manager for R and Python.

I've seen SAS folks respond to posts about Posit by saying, sensibly, "or just get Viya, and you get R and Python and SAS!".

I joined a Posit community call a couple weeks ago. The chat was full of anti-SAS rhetoric, and also plenty of anti-Python sentiment as well. It seems like the R community saw themselves as a SAS-killer a few years ago, but now they feel a bit like Python has overpassed them. Meanwhile, SAS is embracing open-source more than R folks. : )

Re: Databricks vs SAS

Reeza — Wed, 28 Jun 2023 16:32:27 GMT

I'm actually language agnostic, don't really care as long as it gets the job done. The language is used to achieve some business purpose and many different tools can be used. I've used PowerShell for data automation at one point because nothing else was available and it can do a lot surprisingly!

Re: Databricks vs SAS

jimbobob — Thu, 06 Jul 2023 14:11:52 GMT

Half the team uses gui in SAS studio, the other half are straight coders. I would be curious once you get it up and running more detail on the visualizations, since we have many built in SAS, and if were forced to rebuild in Databricks how hard that would be.

Re: Databricks vs SAS

Reeza — Thu, 06 Jul 2023 15:20:00 GMT

Your Visual Analytics visualizations? We have no plans to rebuild those in Databricks, we'll be using PowerBI for those instead.