bachfan Tracker

Stratefication with text topic

bachfan — Mon, 10 Aug 2015 21:59:31 GMT

I am trying to use stratification to simplify the process, but need some help on how to combine the output.

Instead of creating topic nodes for each level of a stratification variable (saying gender), I am wrapping text parser, text filter and text topic nodes within start/end group.

Normally, the output of a topic node will have the topic assignment for each document. But now because the process will go through each level of the stratification variable, I do not see how we can get the output for all the input documents.

Score a dataset with SVM

bachfan — Wed, 17 Sep 2014 22:36:06 GMT

I have trained an SVM model with great success. Now I bring in another dataset for scoring. It has all the attributes except the target variable.

However, I got an error below. Does it mean that SVM scoring function is not provided in SAS EM 12.1? Thank you.

19570 +proc svmscore data= &em_score_output inest=EMWS1.SVM_OUTEST infit=EMWS1.SVM_SVMFITSTAT inclass=EMWS1.SVM_OUTCLASS

19571 + out = &em_score_output(rename=(_F_=F_target_ind _I_=I_target_ind))

19572 +;

19573 +run;

ERROR: Variable Replicate not found on data set.

WARNING: This is an experimental release of the SVMSCORE procedure. It has not had the same level of testing as other components of the SAS system.

NOTE: The SAS System stopped processing this step because of errors. WARNING: The data set EMWS1.SVM_SCORE may be incomplete. When this step was stopped there were 0 observations and 0 variables. WARNING: Data set EMWS1.SVM_SCORE was not replaced because this step was stopped.

Re: How to get TFIDF table from Text Parser

bachfan — Wed, 17 Sep 2014 20:58:19 GMT

Great. I use proc contents and find many more datasets.

But I have a follow-up question. I got great result when using SVM model based on TFIDF matrix as the input variables for classification purposes.

Now I need a scoring dataset, that will go through parsing and filtering. But I do not see a way to get the TFIDF matrix based on the score dataset, which will be subsequently used by SVM. This is because there is only one transaction dataset out of the text filter node. Is this doable?

How to get TFIDF table from Text Parser

bachfan — Tue, 16 Sep 2014 19:28:53 GMT

I am using SAS EM 12.1 version now. I would like to get term frequency inversed document frequency table from Text Parser.

I can see that the exported transaction dataset is the table. It has three column: term index column, document index column, and the weight for that term in that document. However, each term is represented as an index, not the actual word.

Is there a way to find the look up table to map each term's index with the actual word?