Building models with SAS Enterprise Miner, SAS Factory Miner, SAS Visual Data Mining and Machine Learning or just with programming

Scoring data and fit statistics in SAS E Miner

Accepted Solution Solved
Reply
New Contributor
Posts: 2
Accepted Solution

Scoring data and fit statistics in SAS E Miner

[ Edited ]

Hi folks,

I am building a predictive model in SAS E Miner and through model comparison I choosed Neural Network model as best. I have also imported scoring data and scored new data with Neural Network model.I got predicted values and predicted probabilities in a new table. But requirement is also to get the fit statistics for the new scored data.
Please help!!


Accepted Solutions
Solution
a month ago
SAS Employee
Posts: 121

Re: Scoring data and fit statistics in SAS E Miner

[ Edited ]

Please note that to compute assessment statistics, you will need to have the outcome/target information for each observation.   You cannot assess the fit if the outcome is unknown.  In most cases, you are scoring new data where the outcome is unknown.  If you later want to assess how the model performed, you can use a Model Import node to import a scored data set containing predicted values (or predicted probabilities) and compute the assessment statistics.  

 

To do so, follow these steps:

 

1 - Add the scored data as a new Input data source and set the role to Training (even though it is a 'scored' data set)

       Note:  You will need a separate Model Import node flow for each scored data set you wish to import.

 2 - Make sure the target variable has the same name as the one used in the SAS Enterprise Miner modeling flow

 3 - Make sure at least one variable (even a dummy variable) is listed as an Input variable;

 4 - Make sure any binary/nominal/ordinal target variable has the same level and is sorted in the same order as the one used in the SAS Enterprise Miner modeling flow

 5 - In the Model Import node, map the probability of the target event to the correct level of the target variable (if using binary/nominal/ordinal target variable).

 

You can then view the Assessment information in the Model Comparison node.   

 

As Wendy mentioned, if you have the outcome/target information and the input data, you can also specify the data source as Test (assuming you don't already have a Test data set) and pass it to the first node beyond the Data Partition node.  If no Data Partition node is present, connect it to the same node the training and/or validation data set is connected to.  SAS Enterprise Miner will then create assessment information for the 'Test' data (which is actually your scored data) as part of the flow.  

View solution in original post


All Replies
SAS Super FREQ
Posts: 272

Re: Scoring data and fit statistics in SAS E Miner

Can you just clarify for me what your exact steps are?

You have a flow with a Model Comparison node that chooses the Neural Network model.

 

Do you then attach the Model Comparison node and another Input Data node that has new data (but has the observed target in it) to score to a Score node?  If so, what role is assigned to your Input Data?

Then you want assessment for that, correct?

 

Please let me know if I'm understanding correctly!

New Contributor
Posts: 2

Re: Scoring data and fit statistics in SAS E Miner

Hi,

Thanks for reply!

Yes, Model Comparison node  is used in the flow which chooses Neural Network as a best model.

Yes, I attached new input data with the Model Comparison node and used Score node to predict new cases.

Input role is assigned to the input data node.

Yes need a assessment of input data.

SAS Super FREQ
Posts: 272

Re: Scoring data and fit statistics in SAS E Miner

There might be a better way to do this, but the only way I can figure is to set the role of this Input Data node to Test (this assumes you don't already have a Test partition), and connect that directly to the Neural Network node instead of the Score node, and you will get the assessment statistics there.  The Score node is typically used for scoring new data where the target is unknown.

Solution
a month ago
SAS Employee
Posts: 121

Re: Scoring data and fit statistics in SAS E Miner

[ Edited ]

Please note that to compute assessment statistics, you will need to have the outcome/target information for each observation.   You cannot assess the fit if the outcome is unknown.  In most cases, you are scoring new data where the outcome is unknown.  If you later want to assess how the model performed, you can use a Model Import node to import a scored data set containing predicted values (or predicted probabilities) and compute the assessment statistics.  

 

To do so, follow these steps:

 

1 - Add the scored data as a new Input data source and set the role to Training (even though it is a 'scored' data set)

       Note:  You will need a separate Model Import node flow for each scored data set you wish to import.

 2 - Make sure the target variable has the same name as the one used in the SAS Enterprise Miner modeling flow

 3 - Make sure at least one variable (even a dummy variable) is listed as an Input variable;

 4 - Make sure any binary/nominal/ordinal target variable has the same level and is sorted in the same order as the one used in the SAS Enterprise Miner modeling flow

 5 - In the Model Import node, map the probability of the target event to the correct level of the target variable (if using binary/nominal/ordinal target variable).

 

You can then view the Assessment information in the Model Comparison node.   

 

As Wendy mentioned, if you have the outcome/target information and the input data, you can also specify the data source as Test (assuming you don't already have a Test data set) and pass it to the first node beyond the Data Partition node.  If no Data Partition node is present, connect it to the same node the training and/or validation data set is connected to.  SAS Enterprise Miner will then create assessment information for the 'Test' data (which is actually your scored data) as part of the flow.  

☑ This topic is solved.

Need further help from the community? Please ask a new question.

Discussion stats
  • 4 replies
  • 542 views
  • 0 likes
  • 3 in conversation