Feature engineering with autopilot actions, and samplig (surveyselect vs partition procedures)

andrea_magatti — Mon, 07 Mar 2022 15:22:13 GMT

Hi, community,

I want to explain a couple of situations where Viya (3.5) behaves differently from SAS 9.4.

The cases:

dataSciencePilot.featureMachine
Proc partition vs. Proc surveyselect

In the first case, I encountered a strange behavior with a simple dataset of 15k obs and around 120 features.

I added some date vars to the specific input list on a machine with 512GB ram and 80 cores on the first run.

To my surprise, the actionset has used all the available ram and the swap, causing the cas process to be killed by OS (Redhat). After that, I realized that the date vars were not helpful for the model I was going to build, so these vars have been dropped.

With that change, the process took 30 seconds to complete, so I assume that the distribution of the dates determines some issues. My question is: why Viya hasn't provided any warning in the log?

Second case

While sampling a dataset needed for the TSNE analysis, I first used the surveyselect procedure for a stratified sampling approach. By mistake, I added the logical key of the dataset to the BY group.

The procedure log reported (correctly):

ERROR: The number of strata, 14551, is greater than the total sample size, 1456.

That's fine! I recognized my mistake, and once corrected, I got the results I needed.

Then I tried the same (erroneous) approach with the partition procedure, obtaining the same result I encountered with the dataSciencePilot.featureMachine: The actionSet has consumed both the RAM and the SWAP filesystem without warnings in the log.

Could you explain this behavior?

I appreciate any help you can provide.

topic Feature engineering with autopilot actions, and samplig (surveyselect vs partition procedures) in SAS Data Science

Feature engineering with autopilot actions, and samplig (surveyselect vs partition procedures)