topic Re: Does the test data set need to go through the same data preparation just like the training/val s in SAS Data Science

Does the test data set need to go through the same data preparation just like the training/val set?

kewldude — Sat, 11 Nov 2017 12:14:00 GMT

Hi,

Sorry if this sounds like a stup*d question, but I have two sets of data. 1 is I used as a training/validation set that is imported by the import node. It goes through Drop,Filter,Transform and Partition node (60-40 division for training/validation) before getting fed to different models. now I have another data set that is marked a role of Test, does this data set (which is on an import node) need to go through the same Drop/Filter/Transform nodes before I connect it as an input of the models? I actually tried to do that. I copied the Drop Filter and Transform nodes so it will have its own, but the problem is, its showing me a "Must have at least 1 Train or Raw data set(s)." error. I noticed if i change the role of this import node to train, the error will go away but this node needs to have a role of Test because its a test data set.

Re: Does the test data set need to go through the same data preparation just like the training/val s

MikeStockstill — Tue, 21 Nov 2017 16:10:07 GMT

If your flow is using a Data Partition (creating Train, Validate) node that is downstream from Drop, Filter, and Transform Variables nodes, and if you also want to include a separately defined Test data source, then the answer about where to include the Test data source depends on the degree to which the Test data source is already prepared. There are several moving parts that can affect the answer.

If you want some assistance about debugging a specific flow, you might be best served by sending a model package to technical support. Here is a link that you can use to open a track, and then send your model package.

https://support.sas.com/en/technical-support.html

This note tells how to create a model package. It also tells about how to work around a server error. (A model package does not include your data.)

46764 - Creating a model package (.SPK file) in SAS® Enterprise Miner(tm)
http://support.sas.com/kb/46/764.html