Hi, I am SAS beginner. I need your help with simple question. I need to use my own metric - Log Loss Metric. If I want to use the Model Comparison node, I can't see this type of metric in the options, but I need it. I would like to implement the following: First: Model Comparison node select the best model according Log Loss Metric Second: SAS Model e.g. Gradient boosting, will use Log Loss metric to learn (e.g. R enables to set custom metric and use it in xgboost model to train model) So far I know how to compute Log Loss Metric in SAS Code node. Based on this article An Overview of Machine Learning with SAS® Enterprise Miner™ I implemented this code in Kaggle competition San Francisco Crime classification DATA &EM_EXPORT_TRAIN;
set &EM_IMPORT_DATA end=eof;
array posteriorProbs(1:39) P_CategoryARSON P_CategoryASSAULT P_CategoryBAD_CHECKS P_CategoryBRIBERY P_CategoryBURGLARY P_CategoryDISORDERLY_CONDUCT P_CategoryDRIVING_UNDER_THE_INFL P_CategoryDRUG_NARCOTIC P_CategoryDRUNKENNESS P_CategoryEMBEZZLEMENT P_CategoryEXTORTION P_CategoryFAMILY_OFFENSES P_CategoryFORGERY_COUNTERFEITING P_CategoryFRAUD P_CategoryGAMBLING P_CategoryKIDNAPPING P_CategoryLARCENY_THEFT P_CategoryLIQUOR_LAWS P_CategoryLOITERING P_CategoryMISSING_PERSON P_CategoryNON_CRIMINAL P_CategoryOTHER_OFFENSES P_CategoryPORNOGRAPHY_OBSCENE_MA P_CategoryPROSTITUTION P_CategoryRECOVERED_VEHICLE P_CategoryROBBERY P_CategoryRUNAWAY P_CategorySECONDARY_CODES P_CategorySEX_OFFENSES_FORCIBLE P_CategorySEX_OFFENSES_NON_FORCI P_CategorySTOLEN_PROPERTY P_CategorySUICIDE P_CategorySUSPICIOUS_OCC P_CategoryTREA P_CategoryTRESPASS P_CategoryVANDALISM P_CategoryVEHICLE_THEFT P_CategoryWARRANTS P_CategoryWEAPON_LAWS;
if F_Category = "ARSON" then t=1;
if F_Category = "ASSAULT" then t=2;
if F_Category = "BAD CHECKS" then t=3;
if F_Category = "BRIBERY" then t=4;
if F_Category = "BURGLARY" then t=5;
if F_Category = "DISORDERLY CONDUCT" then t=6;
if F_Category = "DRIVING UNDER THE INFLUENCE" then t=7;
if F_Category = "DRUG/NARCOTIC" then t=8;
if F_Category = "DRUNKENNESS" then t=9;
if F_Category = "EMBEZZLEMENT" then t=10;
if F_Category = "EXTORTION" then t=11;
if F_Category = "FAMILY OFFENSES" then t=12;
if F_Category = "FORGERY/COUNTERFEITING" then t=13;
if F_Category = "FRAUD" then t=14;
if F_Category = "GAMBLING" then t=15;
if F_Category = "KIDNAPPING" then t=16;
if F_Category = "LARCENY/THEFT" then t=17;
if F_Category = "LIQUOR LAWS" then t=18;
if F_Category = "LOITERING" then t=19;
if F_Category = "MISSING PERSON" then t=20;
if F_Category = "NON-CRIMINAL" then t=21;
if F_Category = "OTHER OFFENSES" then t=22;
if F_Category = "PORNOGRAPHY/OBSCENE MAT" then t=23;
if F_Category = "PROSTITUTION" then t=24;
if F_Category = "RECOVERED VEHICLE" then t=25;
if F_Category = "ROBBERY" then t=26;
if F_Category = "RUNAWAY" then t=27;
if F_Category = "SECONDARY CODES" then t=28;
if F_Category = "SEX OFFENSES FORCIBLE" then t=29;
if F_Category = "SEX OFFENSES NON FORCIBLE" then t=30;
if F_Category = "STOLEN PROPERTY" then t=31;
if F_Category = "SUICIDE" then t=32;
if F_Category = "SUSPICIOUS OCC" then t=33;
if F_Category = "TREA" then t=34;
if F_Category = "TRESPASS" then t=35;
if F_Category = "VANDALISM" then t=36;
if F_Category = "VEHICLE THEFT" then t=37;
if F_Category = "WARRANTS" then t=38;
if F_Category = "WEAPON LAWS" then t=39;
retain logloss 0;
logloss+log(posteriorProbs[t]);
if eof then do;
logloss=(-1*logloss)/_N_;
PUT "result " logloss;
end;
RUN; But I don't know how to transmit this result to the Model Comparison Node or how to put it into the Model Node. Maybe it is not possible. This metric is the basic/necessary in modeling and predictions. Is there any other solution to the problem except my approach? I hope that there is more sofisticated approach to the problem than mine. Thanks for help.
... View more
