topic Re: Sas Viya 3.5 Sizing and VDMML in Administration and Deployment

Sas Viya 3.5 Sizing and VDMML

C_Golen — Fri, 14 Apr 2023 14:14:48 GMT

Greetings,

I have 55m lines of data in SAS Viya 3.5. I want to work with this data in vdmml. How is SAS Viya in this regard? How should the specs be on average? ( There are 4 co-workers in our environment. )

Re: Sas Viya 3.5 Sizing and VDMML

sbxkoenk — Fri, 21 Apr 2023 21:02:15 GMT

I have moved your question to the "Administration & Deployment"-board !

55E6 lines is not telling us a lot of course! One column or 2000 columns? Record length?
What is the size of the data table?

Koen

Re: Sas Viya 3.5 Sizing and VDMML

SASKiwi — Sat, 22 Apr 2023 23:35:07 GMT

Tables with 55M rows aren't large. I deal with tables of this size a lot and that is in SAS 9.4 and using the V9 SAS data engine. Why do you think your use case needs special treatment?

Re: Sas Viya 3.5 Sizing and VDMML

C_Golen — Sun, 23 Apr 2023 21:54:52 GMT

Thanks for answering and help about "correct topic"!

You made a good point. There are 208 columns in the data and there are 172 string columns with an average of varchar(70). Here, numeric conversion can be applied to these columns and the data size can be reduced. My main concern here is that when I pull the data into the Viya environment, it appears 46.8 gb and this process takes about 1-3 hour. Based on this, I was worried about whether a ml model would be performant or not.

Re: Sas Viya 3.5 Sizing and VDMML

C_Golen — Sun, 23 Apr 2023 21:57:32 GMT

In our environment SAS 9.4 for ML Models is not applicable. As I mentioned above (Answer to Koen) it contains aprx. 200 columns and most of them string. I have not experience before with SAS Viya with this amount of data.
Did you handle tables of that size for ML purposes? My main goal is predicting the quantity in this dataset. (Transactional sales data)

Re: Sas Viya 3.5 Sizing and VDMML

SASKiwi — Sun, 23 Apr 2023 22:14:43 GMT

Sorry, I don't have any firsthand experience in VDMML. However what is the context of your question? Do you already have a Viya 3.5 installation and wish to know if it will scale for VDMML or are you planning to install it and want advice on your initial sizing? In the second case, SAS offer a sizing service which I've found very useful in the past.

If you already have a Viya 3.5 installation, then you can easily run some ML tests to see how it scales with increasing data volumes.

Re: Sas Viya 3.5 Sizing and VDMML

C_Golen — Wed, 26 Apr 2023 07:43:14 GMT

Hi again.

We've already have Viya. the best approach "try-fail", i guess. Thx for your contributions!