Sas Viya 3.5 Sizing and VDMML

[C_Golen]

Greetings,

I have 55m lines of data in SAS Viya 3.5. I want to work with this data in vdmml. How is SAS Viya in this regard? How should the specs be on average? ( There are 4 co-workers in our environment. )

[sbxkoenk]

I have moved your question to the "Administration & Deployment"-board !

55E6 lines is not telling us a lot of course! One column or 2000 columns? Record length?
What is the size of the data table?

Koen

[C_Golen]

Thanks for answering and help about "correct topic"!

You made a good point. There are 208 columns in the data and there are 172 string columns with an average of varchar(70). Here, numeric conversion can be applied to these columns and the data size can be reduced. My main concern here is that when I pull the data into the Viya environment, it appears 46.8 gb and this process takes about 1-3 hour. Based on this, I was worried about whether a ml model would be performant or not.

[SASKiwi]

Tables with 55M rows aren't large. I deal with tables of this size a lot and that is in SAS 9.4 and using the V9 SAS data engine. Why do you think your use case needs special treatment?

[C_Golen]

In our environment SAS 9.4 for ML Models is not applicable. As I mentioned above (Answer to Koen) it contains aprx. 200 columns and most of them string. I have not experience before with SAS Viya with this amount of data.
Did you handle tables of that size for ML purposes? My main goal is predicting the quantity in this dataset. (Transactional sales data)

[SASKiwi · Accepted solution]

Sorry, I don't have any firsthand experience in VDMML. However what is the context of your question? Do you already have a Viya 3.5 installation and wish to know if it will scale for VDMML or are you planning to install it and want advice on your initial sizing? In the second case, SAS offer a sizing service which I've found very useful in the past.

If you already have a Viya 3.5 installation, then you can easily run some ML tests to see how it scales with increasing data volumes. 

[C_Golen]

Hi again.

We've already have Viya. the best approach "try-fail", i guess. Thx for your contributions!