About M_Maldonado

Anotherdream · ‎10-02-2014

Ah astounding you beat me right to it! I also think what you are asking for is exactly accomplished by Astounding... I actually came to the exact same solution, a little late tho.

M_Maldonado · ‎10-03-2014

Fabio, I added a detailed example this morning on how to adjust probabilities for a 50/50 oversample using a Decisions node. Take a look here:. I hope it helps, Miguel

M_Maldonado · ‎09-29-2014

Hi Hussein, What do you mean by "type of cross-validation" in neural networks? The Start/End group nodes support cross validation for any model. For a quick example, take a look at page 7 of this paper (http://support.sas.com/resources/papers/proceedings10/123-2010.pdf)... I can't remember top of my head if you have that option in EM 6.2. If your EM version does not have cross validation option for the Start groups, I suggest go back to the basics. When I read the wikipedia definition of k-fold validation, it sounds like something easy to do in EM. Just an idea, use several Sample nodes, add the same model node to all of them, and use an Ensemble node to combine the posterior predicted probabilities. I have not ran this, but that would be my first try. Please let me know if this worked OK. Thanks! Miguel

M_Maldonado · ‎09-24-2014

Hi Jon, Maybe I am overthinking what you mean by "rules for the final model by majority vote". Where did you get that idea or definition? As far as I understand there is no voting of the rules on a bagging model, but on the predicted probabilities of each of those models. In your specific example, every observation in your data set will be scored with each of one of the trees in your bagged model. The predicted probabilities of these models are then averaged. Below a diagram to illustrate what happens behind the scenes (I grabbed it from the paper I mentioned). The actual code in the Start Groups and End Groups nodes is quite more efficient. Using this alternative diagram you could switch the option in the Ensemble node from averaging to vote. But again, you are voting on the predicted probabilities of each of the models for an observation, not voting on the rules of the models. Whether you combine the predicted probabilities of the models by averaging or by voting, it is the predicted (posterior) probabilities, not the rules. When you have data partition, your models are assessed using a statistic of your validation set. For the specific case of trees, the pruning of the tree model will be based on the validation set. Are you sure it is not the rules of each model that you want to see? For example, in the figure I pasted you have 4 Decision Trees. You could learn more details of your model by looking at the rules of each tree if you think that adds value, but you cannot combine them into "rules of the final model". Does this help at all? Miguel

M_Maldonado · ‎09-17-2014

Hi, In a bunch of places Here 3 that I can think top of my mind. 1. Results Output On the results of a model node, go to the output and punch the keys Ctrl+F to find "Event Classification Table". You will see one like the below: Event Classification Table Data Role=TRAIN Target=BAD False True False True Negative Negative Positive Positive 364 4518 253 825 2. Classification chart If you are a visual person, see this in a chart! After you run a model node, go to results, then browse to View->Assessment->Classification chart. This gives you a stacked bar chart of correct vs incorrect predicted target. To see it in a table format, click on the table icon (2nd icon right to left). You will have all the info you need (target, outcome, target percentage, total percentage, etc). 3. Model comparison node When your diagram has a categorical target variable, the model comparison will use misclassification as the default selection statistic. You will find the Event Classificaiton Table of all models connected to that model comparison node. Was this a good answer? Good luck! -Miguel

Ksharp · ‎09-16-2014

I will Winsorted . Replace them with 90th or 95th percentile .

M_Maldonado · ‎07-18-2014

You can also use proc export in a SAS Code node. Paste the below on the code editor, or change &EM_EXPORT_TRAIN for the macro variable of the data set that you are interested. proc export data=&EM_EXPORT_TRAIN outfile='c:\myfiles\mymodel.csv' dbms=csv replace; run;

Reeza · ‎06-10-2015

I mixed up K-Means and KNN.

omerzeybek · ‎08-01-2014

Hi Miguel here i pasted log file i recevied... when "USE DESICION" is OK Thank you very much 50597 + %let _segment = &segment; 50598 + %let _samplerate = &samplerate; 50599 + %let _useexternal = &useexternal; 50601 + filename _x catalog 'sashelp.emmeta.emadvisecolumns.source'; 50602 + %inc _x; 50603 + filename _x; 50605 + %if &wip_ds_columnmeta eq %then 50606 + %let wip_ds_columnmeta = WORK.COLUMNMETA; 50608 + %if &_computeStat eq Y and &wip_ds_statistic eq %then 50609 + %let wip_ds_statistic = WORK.STATISTIC; 50611 + %EmAdviseColumns(&wip_ds_data, applyIntervalLevelLowerLimit= &_applyIntervalLevelLowerLimit, 50612 + applyMaxClassLevels = &_applyMaxClassLevels, applyMaxPercentMissing = &_applyMaxPercentMissing, 50613 + identifyEmptyColumns = &_identifyEmptyColumns, intervalLevelLowerLimit = &_intervalLevelLowerLimit, 50614 + maxClassLevels = &_maxClassLevels, maxPercentMissing = &_maxPercentMissing, computeStat = &_computeStat, 50615 + outdata=&wip_ds_columnmeta, statisticData=&wip_ds_statistic); 50616 + %end; 50617 + dm "afa c=sashelp.emwip.em_datasource.scl"; 50618 + %end; 50619 + %else 50620 + %if "&WIP_DS_ACTION"="DELETE" %then %do; 50621 + %if "&dataSourceId" eq "" or ^%sysfunc(exist(&DS_LIB..&dataSourceId._p)) %then %do; 50622 + %put DataSource ID has not been specified or is invalid.; 50623 + %end; 50624 + %else %do; 50625 + dm "afa c=sashelp.emwip.em_datasource.scl"; 50626 + %end; 50627 + %end; 50628 + %else 50629 + %if "&WIP_DS_ACTION" ="GETDATASOURCES" %then %do; 50631 + dm "afa c=sashelp.emwip.em_datasource.scl"; 50633 + %end; 50634 + %else 50635 + %if "&WIP_DS_ACTION" ="GETDATASOURCEINFO" %then %do; 50636 + %let memberId = &dataSourceid; 50638 + %if "&memberId" eq "" or ^%sysfunc(exist(&DS_LIB..&memberId._p)) %then %do; 50639 + %put DataSource ID has not been specified or is invalid.; 50640 + %end; 50641 + %else %do; 50642 + dm "afa c=sashelp.emwip.em_datasource.scl"; 50643 + %end; 50644 + %end; 50645 + %else 50646 + %if "&WIP_DS_ACTION" ="RENAME" %then %do; 50647 + %let memberId = &dataSourceid; 50648 + %if "&memberId" eq "" or ^%sysfunc(exist(&DS_LIB..&memberId._p)) %then %do; 50649 + %put DataSource ID has not been specified or is invalid.; 50650 + %end; 50651 + %else %do; 50652 + dm "afa c=sashelp.emwip.em_datasource.scl"; 50653 + %end; 50654 + %end; 50655 + %if "&WIP_DS_ACTION" ="REFRESH" %then %do; 50656 + %let datasourcePath = %sysfunc(pathname(&DS_LIB)); 50657 + %let memberId = &dataSourceid; 50659 + %if "&memberId" eq "" or ^%sysfunc(exist(&DS_LIB..&memberId._p)) %then %do; 50660 + %put DataSource ID has not been specified or is invalid.; 50661 + %end; 50662 + %else %do; 50663 + %let _cmds = &DS_LIB..&memberId._cm; 50664 + %let _tmds = &DS_LIB..&memberId._tm; 50666 + %let EM_DATA=; 50667 + data _null_; 50668 + set &_tmds; 50669 + call symput('EM_DATA', strip(libname)!!'.'!!strip(memname)); 50670 + run; 50672 + %EmRefreshMeta(dataset=&EM_DATA, ColumnsMeta=&_cmds, TableMeta=&_tmds); 50674 + data &DS_LIB..&memberId._P; 50675 + set &DS_LIB..&memberId._p; 50676 + if Property='ModifyDate' then Value= strip(put(datetime(), best.)); 50677 + else 50678 + if Property='ModifiedBy' then Value = "&sysuserid"; 50679 + run; 50680 + dm "afa c=sashelp.emwip.em_datasource.scl"; 50681 + %end; 50682 + %end; 50683 + %else 50684 + %if "&WIP_DS_ACTION" ="COPY" %then %do; 50685 + %let memberId = &dataSourceid; 50686 + %if "&memberId" eq "" or ^%sysfunc(exist(&DS_LIB..&memberId._p)) %then %do; 50687 + %put DataSource ID has not been specified or is invalid.; 50688 + %end; 50689 + %else %do; 50690 + dm "afa c=sashelp.emwip.em_datasource.scl"; 50691 + %end; 50692 + %end; 50693 + %if "&library" eq "" and ^%sysfunc(libref(&DS_LIB)) %then %do; 50694 + %if "&DS_LIB" ne "WORK" %then 50695 + libname &DS_LIB; 50696 + %end; 50698 +%mend em_dataSource; NOTE: %INCLUDE (level 2) ending. NOTE: %INCLUDE (level 1) resuming. 50699 + filename _temp; NOTE: Fileref _TEMP has been deassigned. 50700 + filename _temp catalog 'sashelp.emwip.em_init.source'; 50701 + %include _temp; NOTE: %INCLUDE (level 2) file _TEMP is file SASHELP.EMWIP.EM_INIT.SOURCE. 50702 +%macro em_init(projpath=, projname= ); 50704 + %global wip_projpath; 50705 + %global wip_projectname _emdsep; 50706 + %do; 50708 + %let wip_projpath = %superq(projpath); 50709 + %let wip_projectname = %superq(projname); 50711 + %if ^%symexist(EM_DEBUG) %then %do; 50712 + %global EM_DEBUG; 50713 + %let EM_DEBUG=; 50714 + %end; 50715 + %if ^%symexist(EMEXCEPTIONSTRING) %then %do; 50716 + %global EMEXCEPTIONSTRING; 50717 + %let EMEXCEPTIONSTRING=; 50718 + %end; 50720 + dm "afa c=sashelp.emwip.em_init.scl"; 50721 + %end; 50723 +%mend em_init; NOTE: %INCLUDE (level 2) ending. NOTE: %INCLUDE (level 1) resuming. 50724 + filename _temp; NOTE: Fileref _TEMP has been deassigned. 50725 + filename _temp catalog 'sashelp.emwip.em_project.source'; 50726 + %include _temp; NOTE: %INCLUDE (level 2) file _TEMP is file SASHELP.EMWIP.EM_PROJECT.SOURCE. 50727 +/* 50728 + MACRO: EM_PROJECT 50730 + PURPOSE: Interact with an EM Project 50732 + PARAMETERS: 50734 + ACTION = requested action (CREATE, ADDDIAGRAM, DELETE, DELETEDIAGRAM, GETDIAGRAMS, GETDIAGRAMINFO) 50735 + PROJPATH = location of the project 50736 + PROJNAME = project name 50737 + DGMNAME = diagram name (needed when DELETEDIAGRAM or GETDIAGRAMINFO is called) 50738 + OUTPUT = output data set (when requesting information) 50739 +*/ 50741 +%macro wip_decode(macvar=, string=); 50742 + %let string= %qsysfunc(tranwrd(%superq(string), %nrstr(&), %nrstr(&))); 50743 + %let string= %qsysfunc(tranwrd(%superq(string), %nrstr(&pct;), %)); 50744 + %let string= %qsysfunc(tranwrd(%superq(string), %nrstr('), %str(%'))); 50745 + %let string= %qsysfunc(tranwrd(%superq(string), %nrstr("), %str(%"))); 50746 + %let string= %qsysfunc(tranwrd(%superq(string), %nrstr(&lparen;), %str(%())); 50747 + %let string= %qsysfunc(tranwrd(%superq(string), %nrstr(&rparen;), %str(%)))); 50748 + %let &macvar = %superq(string); 50749 +%mend wip_decode; 50751 +%macro em_project(action=create, projpath=, projname=, userid=, dgmId=, dgmName=, miningresultid=, 50752 + requestFile=REQUEST.XML, outfile=RESPONSE.XML); 50753 + filename _x catalog 'sashelp.emwip.em_dsep.source'; 50754 + %inc _x; 50755 + filename _x; 50756 + %em_dsep; 50758 + %global emexceptionString em_project_exist; 50759 + %let EMEXCEPTIONSTRING=; 50760 + %let em_project_exist= N; 50762 + %global wip_action wip_projpath wip_projectname wip_dgmid wip_dgmname wip_outfile 50763 + wip_requestfile wip_mrid _wiprc wip_userid; 50765 + %let WIP_ACTION = &action; 50767 + %wip_decode(macvar=WIP_PROJPATH, string=%superq(projpath)); 50768 + %wip_decode(macvar=WIP_PROJECTNAME, string=%superq(projname)); 50770 + %if "&WIP_userid" eq "" %then %do; 50771 + %let WIP_userid = %superq(userId); 50772 + %if "&WIP_userid" = "" %then 50773 + %let WIP_userid = %superq(sysuserid); 50774 + %end; 50776 + %let WIP_DGMID = &dgmId; 50777 + %wip_decode(macvar=WIP_DGMNAME, string=%superq(dgmName)); 50779 + %let WIP_OUTFILE = &outfile; 50780 + %let WIP_REQUESTFILE = %superq(requestFile); 50782 + %let WIP_MRID = &miningresultid; 50784 + %let EM_ACTION = %upcase(&action); 50785 + %if "&EM_ACTION" ="EXIST" %then %do; 50787 + %if %sysfunc(fileexist(&WIP_PROJPATH&emdsep&WIP_PROJECTNAME&emdsep.Workspaces)) and 50788 + %sysfunc(fileexist(&WIP_PROJPATH&emdsep&WIP_PROJECTNAME&emdsep.DataSources))and 50789 + %sysfunc(fileexist(&WIP_PROJPATH&emdsep&WIP_PROJECTNAME&emdsep.Meta)) and 50790 + %sysfunc(fileexist(&WIP_PROJPATH&emdsep&WIP_PROJECTNAME&emdsep.Reports)) and 50791 + %sysfunc(fileexist(&WIP_PROJPATH&emdsep&WIP_PROJECTNAME&emdsep.System)) and 50792 + %sysfunc(fileexist(&WIP_PROJPATH&emdsep&WIP_PROJECTNAME&emdsep.project.emp)) %then %do; 50793 + %let em_project_exist = Y; 50794 + %end; 50795 + %end; 50796 + %else 50797 + %if "&EM_ACTION" ="CREATE" %then %do; 50798 + %if %sysfunc(fileexist(&WIP_PROJPATH&emdsep&WIP_PROJECTNAME&emdsep.Workspaces)) and 50799 + %sysfunc(fileexist(&WIP_PROJPATH&emdsep&WIP_PROJECTNAME&emdsep.DataSources))and 50800 + %sysfunc(fileexist(&WIP_PROJPATH&emdsep&WIP_PROJECTNAME&emdsep.Meta)) and 50801 + %sysfunc(fileexist(&WIP_PROJPATH&emdsep&WIP_PROJECTNAME&emdsep.Reports)) and 50802 + %sysfunc(fileexist(&WIP_PROJPATH&emdsep&WIP_PROJECTNAME&emdsep.System)) and 50803 + %sysfunc(fileexist(&WIP_PROJPATH&emdsep&WIP_PROJECTNAME&emdsep.project.emp)) %then %do; 50804 + %let em_project_exist = Y; 50805 + %put Project already exists.; 50806 + %end; 50807 + %else %do; 50808 + dm "afa c=sashelp.emwip.em_project.scl"; 50809 + %end; 50810 + %end; 50811 + %else 50812 + %if "&EM_ACTION" ="SETPROPERTIES" %then %do; 50813 + dm "afa c=sashelp.emwip.em_project.scl"; 50814 + %end; 50815 + %else 50816 + %if "&EM_ACTION" = "GETPROPERTIES" %then %do; 50817 + dm "afa c=sashelp.emwip.em_project.scl"; 50818 + %end; 50819 + %else 50820 + %if "&EM_ACTION" ="RENAMEDIAGRAM" %then %do; 50821 + dm "afa c=sashelp.emwip.em_project.scl"; 50822 + %end; 50823 + %else 50824 + %if "&EM_ACTION" ="ADDDIAGRAM" %then %do; 50825 + %if ^(%sysfunc(fileexist(&WIP_PROJPATH&emdsep&WIP_PROJECTNAME&emdsep.project.emp))) %then %do; 50826 + %put Project does not exists.; 50827 + %end; 50828 + %else %do; 50829 + dm "afa c=sashelp.emwip.em_project.scl"; 50830 + %end; 50831 + %end; 50832 + %else 50833 + %if "&EM_ACTION" ="IMPORTDIAGRAM" %then %do; 50834 + %if ^(%sysfunc(fileexist(&WIP_PROJPATH&emdsep&WIP_PROJECTNAME&emdsep.project.emp))) %then %do; 50835 + %put Project does not exists.; 50836 + %end; 50837 + %else %do; 50838 + proc display c=sashelp.emwip.em_project.scl;run; 50839 + %end; 50840 + %end; 50841 + %else 50842 + %if "&EM_ACTION" ="DELETE" %then %do; 50843 + proc display c=sashelp.emwip.em_project.scl;run; 50845 + %let _WIPRC = 0; 50846 + %if %sysfunc(fileexist(&WIP_PROJPATH&emdsep&WIP_PROJECTNAME))%then %do; 50847 + %let _WIPRC = 1; 50848 + %end; 50850 + %end; 50851 + %else 50852 + %if "&EM_ACTION" ="DELETEDIAGRAM" %then %do; 50853 + %if "&wip_dgmId" ne "" %then %do; 50854 + dm "afa c=sashelp.emwip.em_project.scl"; 50855 + %end; 50856 + %else %do; 50857 + %put Diagram ID not specified.; 50858 + %end; 50859 + %end; 50860 + %else %do; 50861 + dm "afa c=sashelp.emwip.em_project.scl"; 50862 + %end; 50863 +%mend em_project; NOTE: %INCLUDE (level 2) ending. NOTE: %INCLUDE (level 1) resuming. 50864 + filename _temp; NOTE: Fileref _TEMP has been deassigned. 50865 + filename _temp catalog 'sashelp.emwip.em_diagram.source'; 50866 + %include _temp; NOTE: %INCLUDE (level 2) file _TEMP is file SASHELP.EMWIP.EM_DIAGRAM.SOURCE. 50869 +%macro wip_decode(macvar=, string=); 50870 + %let string= %qsysfunc(tranwrd(%superq(string), %nrstr(&), %nrstr(&))); 50871 + %let string= %qsysfunc(tranwrd(%superq(string), %nrstr(&pct;), %)); 50872 + %let string= %qsysfunc(tranwrd(%superq(string), %nrstr('), %str(%'))); 50873 + %let string= %qsysfunc(tranwrd(%superq(string), %nrstr("), %str(%"))); 50874 + %let string= %qsysfunc(tranwrd(%superq(string), %nrstr(&lparen;), %str(%())); 50875 + %let string= %qsysfunc(tranwrd(%superq(string), %nrstr(&rparen;), %str(%)))); 50876 + %let &macvar = %superq(string); 50877 +%mend wip_decode; 50879 +%macro em_diagram(action=, projpath=, projname=, dgmId=, sessionid=, userid=, nodeid=, from=, to=, component=, x=-1, y=-1, 50880 + toolprefix=, tooltype=,toolcode=, numtask=SINGLE, sascmd=, datasource=, serverclass=, 50881 + datakey=, outfile=, reportname=, requestfile=, mode=, runkey=, socket=, url=, javapackagename=, jscore=, cscore= , 50882 + columnmeta=, tablemeta=, statistic=, data=, sample=, orientation=, serverStartUpCode=); 50884 + filename _dgmfrf catalog 'sashelp.emwip.em_dsep.source'; 50885 + %inc _dgmfrf; 50886 + filename _dgmfrf; 50887 + %em_dsep; 50889 + %global emexceptionString; 50890 + %let emexceptionstring=; 50892 + %global wip_prefix wip_nodeid wip_fromid wip_toid wip_component wip_x wip_y 50893 + wip_toolprefix wip_tooltype wip_toolcode wip_projpath wip_projectname wip_emlib wip_action 50894 + wip_datasource wip_numtask wip_sessionid wip_outfile wip_requestfile wip_datakey 50895 + wip_mode wip_runkey wip_socket wip_url wip_userid wip_cscore wip_jscore wip_jpackage WIP_serverclass 50896 + wip_data wip_columnmeta wip_tablemeta wip_statistic wip_sample WIP_orientation WIP_serverStartUpCode; 50898 + %wip_decode(macvar=WIP_PROJPATH, string=%superq(projpath)); 50899 + %wip_decode(macvar=WIP_PROJECTNAME, string=%superq(projname)); 50901 + %let WIP_EMLIB = &dgmId; 50902 + %let WIP_PREFIX = &toolprefix; 50903 + %let WIP_NODEID = &nodeid; 50904 + %let WIP_FROMID = &from; 50905 + %let WIP_TOID = &to; 50906 + %let WIP_COMPONENT = &component; 50907 + %let WIP_TOOLPREFIX = &toolprefix; 50908 + %let WIP_TOOLTYPE = &tooltype; 50909 + %let WIP_TOOLCODE = &toolcode; 50910 + %let WIP_X = &x; 50911 + %let WIP_Y = &y; 50912 + %let WIP_DATASOURCE = &datasource; 50913 + %let WIP_numtask = &numtask; 50914 + %let WIP_sascmd = &sascmd; 50915 + %let WIP_reportName = %superq(reportName); 50916 + %let WIP_outfile = %nrbquote(&outfile); 50917 + %let WIP_requestFile = %nrbquote(&requestFile); 50918 + %let WIP_datakey = %nrbquote(&datakey); 50919 + %let WIP_mode = &mode; 50920 + %let WIP_runkey = &runkey; 50921 + %let WIP_socket = &socket; 50922 + %let WIP_URL = &url; 50923 + %let WIP_jscore = &jscore; 50924 + %let WIP_Cscore = &cscore; 50925 + %let WIP_jpackage = %nrbquote(&javapackagename); 50926 + %let WIP_serverclass = %nrbquote(&serverclass); 50927 + %let WIP_columnmeta = &columnmeta; 50928 + %let WIP_tablemeta = &tablemeta; 50929 + %let WIP_statistic = &statistic; 50930 + %let WIP_data = &data; 50931 + %let WIP_sample = &sample; 50932 + %let WIP_orientation = &orientation; 50933 + %let WIP_serverStartUpCode = &serverStartUpCode; 50935 + %if "&WIP_sessionid" eq "" %then %do; 50936 + %let WIP_sessionid = &sessionId; 50937 + %end; 50939 + %if "&WIP_userid" eq "" %then %do; 50940 + %let WIP_userid = %superq(userId); 50941 + %if "&WIP_userid" = "" %then 50942 + %let WIP_userid = %superq(sysuserid); 50943 + %end; 50945 + %if "&dgmId" eq "" %then 50946 + %goto doendgm; 50948 + %let dgmFolder = %superq(WIP_PROJPATH)&emdsep%superq(WIP_PROJECTNAME)&emdsep.Workspaces&emdsep&WIP_emlib; 50949 + %if ^%sysfunc(fileexist(&dgmFolder)) %then %do; 50950 + %let EMEXCEPTIONSTRING= exception.server.WORKSPACE.CANTOPENWORKSPACE,&dgmId; 50951 + %goto doendgm; 50952 + %end; 50954 + %put WIP_ACTION: &WIP_ACTION; 50955 + %put DGMID: &dgmid; 50957 + %let openlockFile = %superq(WIP_PROJPATH)&emdsep%superq(WIP_PROJECTNAME)&emdsep.Workspaces&emdsep&WIP_emlib&emdsep.System&emdsep.wsopen.lck; 50958 + %put LOCKFILE: &openlockfile; 50960 + %let WIP_ACTION = %upcase(&action); 50961 + %if "&WIP_ACTION" ="INITSESSION" %then %do; 50962 + %if ^%sysfunc(fileexist(&openlockfile)) %then %do; 50963 + %let EMEXCEPTIONSTRING= exception.server.WORKSPACE.NOTOPEN; 50964 + %goto doendgm; 50965 + %end; 50967 + filename _dgmfrf catalog 'sashelp.emwip.em_init.source'; 50968 + %inc _dgmfrf; 50969 + filename _dgmfrf; 50971 + %em_init(projpath=&WIP_projpath, projname= &WIP_projectname); 50973 + libname &wip_emlib "%bquote(&WIP_PROJPATH)&emdsep%bquote(&WIP_PROJECTNAME)&emdsep.Workspaces&emdsep&WIP_emlib"; 50974 + %end; 50975 + %else 50976 + %if "&WIP_ACTION" ="OPENSESSION" %then %do; 50977 + %if ^%sysfunc(fileexist(&openlockfile)) %then %do; 50978 + %let EMEXCEPTIONSTRING= exception.server.WORKSPACE.NOTOPEN; 50979 + %goto doendgm; 50980 + %end; 50982 + libname &wip_emlib "%bquote(&WIP_PROJPATH)&emdsep%bquote(&WIP_PROJECTNAME)&emdsep.Workspaces&emdsep&WIP_emlib"; 50984 + %let dgraph = &wip_emlib..EM_DGRAPH; 50985 + %if %sysfunc(exist(&dgraph)) %then %do; 50986 + data work.EM_DGRAPH; 50987 + set &dgraph; 50988 + run; 50989 + %end; 50991 + dm "afa c=sashelp.emwip.em_diagram.scl"; 50993 + %end; 50994 + %else 50995 + %if "&WIP_ACTION" ="CLOSESESSION" %then %do; 50996 + %if ^%sysfunc(fileexist(&openlockfile)) %then %do; 50997 + %goto doendgm; 50998 + %end; 50999 + libname &wip_emlib; 51001 + %end; 51002 + %else 51003 + %if "&WIP_ACTION" ="OPEN" %then %do; 51004 + %if %sysfunc(fileexist(&openlockfile)) %then %do; 51005 + %let EMEXCEPTIONSTRING= exception.server.WORKSPACE.OPENLOCKED; 51006 + %goto doendgm; 51007 + %end; 51009 + libname &wip_emlib "%bquote(&WIP_PROJPATH)&emdsep%bquote(&WIP_PROJECTNAME)&emdsep.Workspaces&emdsep&WIP_emlib"; 51011 + dm "afa c=sashelp.emwip.em_diagram.scl"; 51013 + %end; 51014 + %else 51015 + %if "&WIP_ACTION" ="GETPROPERTIES" %then %do; 51017 + %if ("&wip_emlib" eq "") or %sysfunc(libref(&wip_emlib)) %then %do; 51018 + %goto doendgm; 51019 + %end; 51021 + %if "&wip_nodeid" eq "" %then %do; 51022 + %let dgraph = &wip_emlib..EM_DGRAPH; 51023 + %if %sysfunc(exist(&dgraph)) %then %do; 51024 + data work.EM_DGRAPH; 51025 + set &dgraph; 51026 + run; 51027 + %end; 51028 + %end; 51030 + dm "afa c=sashelp.emwip.em_diagram.scl"; 51032 + %end; 51033 + %else 51034 + %if "&WIP_ACTION" ="SAVEASXML" %then %do; 51035 + dm "afa c=sashelp.emwip.em_diagram.scl"; 51036 + %end; 51037 + %else 51038 + %if "&WIP_ACTION" ="GETREPORTSMETA" %then %do; 51040 + %if ("&wip_emlib" eq "") or %sysfunc(libref(&wip_emlib)) %then %do; 51041 + %goto doendgm; 51042 + %end; 51044 + dm "afa c=sashelp.emwip.em_diagram.scl"; 51046 + %end; 51047 + %else 51048 + %if "&WIP_ACTION" ="CLOSE" %then %do; 51050 + %if ^%sysfunc(fileexist(&openlockfile)) %then %do; 51051 + %goto doendgm; 51052 + %end; 51053 + dm "afa c=sashelp.emwip.em_diagram.scl"; 51055 + libname &wip_emlib; 51057 + %end; 51058 + %else 51059 + %if "&WIP_ACTION" ="UNLOCK" %then %do; 51061 + %if ^%sysfunc(fileexist(&openlockfile)) %then %do; 51062 + %goto doendgm; 51063 + %end; 51064 + dm "afa c=sashelp.emwip.em_diagram.scl"; 51066 + %end; 51067 + %else 51068 + %if "&WIP_ACTION" ="SETORIENTATION" %then %do; 51070 + %if ^%sysfunc(fileexist(&openlockfile)) %then %do; 51071 + %goto doendgm; 51072 + %end; 51073 + dm "afa c=sashelp.emwip.em_diagram.scl"; 51074 + %end; 51075 + %else 51076 + %if ("&WIP_ACTION" = "COPY") or ("&WIP_ACTION" = "PASTE") %then %do; 51077 + %let runlockFile = &WIP_PROJPATH&emdsep&WIP_PROJECTNAME&emdsep.Workspaces&emdsep&WIP_emlib&emdsep.System&emdsep.wsrun.lck; 51079 + %if ^%sysfunc(fileexist(&openlockfile)) or %sysfunc(fileexist(&runlockfile)) %then %do; 51080 + %goto doendgm; 51081 + %end; 51082 + dm "afa c=sashelp.emwip.em_diagram.scl"; 51084 + %end; 51085 + %else 51086 + %if "&WIP_ACTION" ="RUN" %then %do; 51087 + %let runlockFile = &WIP_PROJPATH&emdsep&WIP_PROJECTNAME&emdsep.Workspaces&emdsep&WIP_emlib&emdsep.System&emdsep.wsrun.lck; 51089 + %if %sysfunc(fileexist(&runlockfile)) or ("&wip_nodeid" eq "") %then %do; 51090 + %goto doendgm; 51091 + %end; 51093 + proc display c=sashelp.emwip.em_diagram.scl batch; 51094 + run; 51095 + %end; 51096 + %else 51097 + %if "&WIP_ACTION" ="MAKEMININGRESULT" %then %do; 51098 + %let runlockFile = &WIP_PROJPATH&emdsep&WIP_PROJECTNAME&emdsep.Workspaces&emdsep&WIP_emlib&emdsep.System&emdsep.wsrun.lck; 51100 + %if %sysfunc(fileexist(&runlockfile)) or ("&wip_nodeid" eq "") %then %do; 51101 + %goto doendgm; 51102 + %end; 51104 + proc display c=sashelp.emwip.em_diagram.scl batch; 51105 + run; 51106 + %end; 51107 + %else %do; 51108 + dm "afa c=sashelp.emwip.em_diagram.scl"; 51110 + %end; 51112 + %doendgm: 51114 +%mend em_diagram; NOTE: %INCLUDE (level 2) ending. NOTE: %INCLUDE (level 1) resuming. 51115 + filename _temp; NOTE: Fileref _TEMP has been deassigned. 51116 + filename _temp catalog 'sashelp.emwip.em_sasgraph.source'; 51117 + %include _temp; NOTE: %INCLUDE (level 2) file _TEMP is file SASHELP.EMWIP.EM_SASGRAPH.SOURCE. 51118 +filename _fdsep catalog 'sashelp.emwip.em_dsep.source'; 51119 +%include _fdsep; NOTE: %INCLUDE (level 3) file _FDSEP is file SASHELP.EMWIP.EM_DSEP.SOURCE. 51120 +%macro em_dsep; 51121 + %global emdsep; 51122 + %if %substr(&sysscp, 1, 3)= WIN %then 51123 + %let emdsep=\; 51124 + %else 51125 + %if %substr(&sysscp, 1, 3)= DNT %then 51126 + %let emdsep=\; 51127 + %else 51128 + %let emdsep=/; 51129 +%mend em_dsep; 51130 +%em_dsep; NOTE: %INCLUDE (level 3) ending. NOTE: %INCLUDE (level 2) resuming. 51131 +filename _fdsep; NOTE: Fileref _FDSEP has been deassigned. 51132 +%em_dsep; 51133 +%macro em_sasgraph(device=JPEG, folder=); 51134 + %if %sysfunc(exist(work.gseg)) %then %do; 51135 + proc delete data=work.gseg; 51136 + run; 51137 + %end; 51138 + proc catalog c=work.gseg; 51139 + contents out=work.gseg; 51140 + run; 51141 + data work.gseg; 51142 + set work.gseg; 51143 + if desc = '' then desc = name; 51144 + run; 51145 + proc sort data=work.gseg; 51146 + by desc; 51147 + run; 51148 + %let _wipGraphPath=%superq(folder); 51149 + %if (%superq(_wipGraphPath) eq ) %then %do; 51150 + data _null_; 51151 + path = pathname('WORK'); 51152 + rc = dcreate('EM_SASGRAPH',strip(path)); 51153 + call symput('_wipGraphPath', strip(path)!!strip(symget('emdsep'))!!'EM_SASGRAPH'); 51154 + run; 51155 + %end; 51156 + filename _dref "&_wipGraphpath"; 51157 + %let did=%sysfunc(dopen(_dref)); 51158 + %let memcount=%sysfunc(dnum(&did)); 51159 + %if &memcount %then %do; 51160 + %do i=1 %to &memcount; 51161 + %let filename=&_wipGraphPath&emdsep%sysfunc(dread(&did, &i)); 51162 + %if "&filename" ne "" %then %do; %put >>>> &filename; 51163 + %let prefix = %sysfunc(reverse(%lowcase(%scan(%nrbquote(%sysfunc(reverse(&filename))), 1, .)))); 51164 + %if (&prefix = jpg) or (&prefix eq gif) %then %do; 51165 + filename _fref "&filename"; 51166 + %let rc = %sysfunc(fdelete(_fref)); 51167 + filename _fref; 51168 + %end; 51169 + %end; 51170 + %end; 51171 + %end; 51172 + %let rc=%sysfunc(dclose(&did)); 51173 + filename _dref; 51174 + data work.gseg; 51175 + keep libname memname name desc type date crdate moddate; 51176 + array token(7) $ ('/' '\' ':' '"' '<' '>' '|'); 51177 + length string $2000 device ftype $8; 51178 + length filename $200; 51179 + retain index 0; 51180 + set work.gseg; 51181 + by desc; 51182 + filename = desc; 51183 + if first.desc then do; 51184 + filename = desc; 51185 + index = 1; 51186 + end; 51187 + else do; 51188 + filename = strip(desc)!!' ('!!strip(put(index, best.))!!')'; 51189 + end; 51190 + do i=1 to 7; 51191 + filename = tranwrd(filename, strip(token(i)), '_'); 51192 + end; 51193 + desc = filename; 51194 + index+1; 51195 + device = upcase("&device"); 51196 + select(device); 51197 + when('JPEG') ftype='jpg'; 51198 + when('GIF') ftype='gif'; 51199 + otherwise do; 51200 + ftype='jpg'; 51201 + device="JPEG"; 51202 + end; 51203 + end; 51204 + string = "&_wipGraphPath&emdsep"!!strip(filename)!!'.'!!ftype; 51205 + call execute('filename gsasfile "'!!strip(string)!!'";'); 51206 + call execute('goptions reset=all device='!!strip(device)!!' display'); 51207 + call execute(' gaccess= gsasfile'); 51208 + call execute(' gsfmode= replace cback=white;'); 51209 + call execute('proc greplay igout=work.gseg nofs;'); 51210 + call execute( 'replay '!!strip(name)!!';'); 51211 + call execute('quit;'); 51212 + call execute('filename gsasfile;'); 51213 + call execute('goptions reset=all device=WMF NODISPLAY;'); 51214 + run; 51215 +%mend em_sasgraph; NOTE: %INCLUDE (level 2) ending. NOTE: %INCLUDE (level 1) resuming. 51220 + filename _temp; NOTE: Fileref _TEMP has been deassigned. NOTE: %INCLUDE (level 1) ending. 51224 filename _emds; NOTE: Fileref _EMDS has been deassigned. 51225 filename _emac catalog 'sashelp.emmeta.emadvisecolumns.source'; 51226 %inc _emac; NOTE: %INCLUDE (level 1) file _EMAC is file SASHELP.EMMETA.EMADVISECOLUMNS.SOURCE. 51227 +/* 51228 + MACRO: EmAdviseColumns 51230 + PURPOSE: Called by datasource wizard 51233 + PARAMETERS: 51234 + none 51236 + USAGE: 51239 + AUTHOR: Paul Tomas 51241 + AMENDMENTS: Initial Version 29Nov01 EURPAT 51242 + Add adviseMode 05Dec01 SASDZL 51243 +---------------------------------------------------------------------- 51244 + D2EMCS61/EM6.1 51245 +---------------------------------------------------------------------- 51246 + S0503797 computeStat, StatisticData 09APR08 SASDZL 51248 +*/ 51251 + %macro EmAdviseColumns(dataset, 51252 + adviseMode=, 51253 + applyIntervalLevelLowerLimit=Y, 51254 + applyMaxClassLevels=Y, 51255 + applyMaxPercentMissing=Y, 51256 + identifyEmptyColumns=Y, 51257 + intervalLevelLowerLimit=20, 51258 + maxLevelDetection=30, 51259 + maxClassLevels = 20, 51260 + maxPercentMissing = 50, 51261 + outdata=WORK.COLUMNMETA, 51262 + computeStat=N, 51263 + statisticData=WORK.STATISTIC, 51264 + passthrough=DEFAULT, 51265 + newdataflag=Y, 51266 + tablemeta=WORK.TABLEMETA, 51267 + DBPassThrough=N); 51268 + %global EM_ADVISENCOUNT; 51269 + %let EM_ADVISENCOUNT=-1; 51270 + proc display c=sashelp.emmeta.advisecolumns.scl; run; 51271 +%mend; NOTE: %INCLUDE (level 1) ending. 51272 filename _emac; NOTE: Fileref _EMAC has been deassigned. ERROR: The submitted line exceeds maximum line length - 32768 bytes. NOTE: The data set WORK.TRANSACTION has 114 observations and 22 variables. NOTE: DATA statement used (Total process time): real time 0.03 seconds cpu time 0.03 seconds 202 The SAS System 10:11 Thursday, July 31, 201 4 51274 %let SYSCC=0; 51275 %let SYSRC=0; 51276 %let EMEXCEPTIONSTRING=; 51277 %let SYSMSG=; 51278 %em_variableeditor(action=CLOSE, projpath=%nrstr(/sasdata/EMiner_Projects), projname=%nrstr(31072014), dgmId=EMWS1, nodeid=VarClus, subset=NONE, port=, transactionmeta=WORK.TRANSACTION, columnmeta=WORK.VARIABLESET); WIP_ACTION: GETATTRIBUTES 51279 data _null_; 51280 set EMWS1.EM_NODEID; 51281 where upcase(NODEID)=upcase("VarClus"); 51282 call symput('_EMCLASS', CLASS); 51283 run; NOTE: There were 1 observations read from the data set EMWS1.EM_NODEID. WHERE UPCASE(NODEID)='VARCLUS'; NOTE: DATA statement used (Total process time): real time 0.00 seconds cpu time 0.00 seconds 51284 proc sort data=WORK.TRANSACTION NOTHREADS; 51285 by NAME; 51286 run; NOTE: There were 114 observations read from the data set WORK.TRANSACTION. NOTE: The data set WORK.TRANSACTION has 114 observations and 22 variables. NOTE: PROCEDURE SORT used (Total process time): real time 0.00 seconds cpu time 0.01 seconds 51287 proc sort data=WORK.VARIABLESET NOTHREADS; 51288 by NAME; 51289 run; NOTE: There were 114 observations read from the data set WORK.VARIABLESET. NOTE: The data set WORK.VARIABLESET has 114 observations and 21 variables. NOTE: PROCEDURE SORT used (Total process time): real time 0.00 seconds cpu time 0.00 seconds 51290 data WORK.VARIABLESET; 51291 drop _transaction_; 51292 update WORK.VARIABLESET(in=_m_) WORK.TRANSACTION; 51293 by NAME; 51294 if ^_m_ then delete; 51295 run; NOTE: There were 114 observations read from the data set WORK.VARIABLESET. NOTE: There were 114 observations read from the data set WORK.TRANSACTION. NOTE: The data set WORK.VARIABLESET has 114 observations and 21 variables. NOTE: DATA statement used (Total process time): real time 0.00 seconds cpu time 0.02 seconds 51296 data EMWS1.VarClus_VariableSet; 51297 set WORK.VARIABLESET; 51298 run; NOTE: There were 114 observations read from the data set WORK.VARIABLESET. NOTE: The data set EMWS1.VARCLUS_VARIABLESET has 114 observations and 21 variables. NOTE: DATA statement used (Total process time): real time 0.00 seconds cpu time 0.00 seconds 51299 %let variableset = WORK.VARIABLESET; PERFORMANCE DETAILS 51518 %let emexceptionString =; 51519 %let EMEXCEPTIONSTRING=; 51520 *------------------------------------------------------------*; 51521 * CLOSEVARIABLESET: VarClus; 51522 *------------------------------------------------------------*; 51523 %let EM_ACTION = CLOSEVARIABLESET; 51524 %let syscc = 0; 51525 51526 %macro main; 51527 51528 filename temp catalog 'sashelp.emexpl.variableclustering_macros.source'; 51529 %include temp; 51530 filename temp catalog 'sashelp.emexpl.variableclustering_macros2.source'; 51531 %include temp; 51532 filename temp; 51533 51534 %SetProperties; 51535 51536 %if %upcase(&EM_ACTION) = CREATE %then %do; 51537 filename temp catalog 'sashelp.emexpl.variableclustering_create.source'; 51538 %include temp; 51539 filename temp; 51540 %create; 51541 %end; 51542 %else 51543 %if %upcase(&EM_ACTION) = TRAIN %then %do; 51544 filename temp catalog 'sashelp.emexpl.variableclustering_train.source'; 51545 %include temp; 51546 filename temp; 51547 %train; 51548 %end; 51549 %else 51550 %if %upcase(&EM_ACTION) = SCORE %then %do; 51551 filename temp catalog 'sashelp.emexpl.variableclustering_score.source'; 51552 %include temp; 51553 filename temp; 51554 %score; 51555 %end; 51556 %else 51557 %if %upcase(&EM_ACTION) = REPORT %then %do; 51558 filename temp catalog 'sashelp.emexpl.variableclustering_report.source'; 51559 %include temp; 51560 filename temp; 51561 %report; 51562 %end; 51563 /* 51564 %if %upcase(&EM_ACTION) = OPENTESTTABLE %then %do; 51565 %put 'OPENING TABLE'; 51566 %end; 51567 %if %upcase(&EM_ACTION) = CLOSETESTTABLE %then %do; 51568 %put 'CLOSE TABLE'; 51569 %end; 51570 */ 51571 %mend main; 51572 %main; NOTE: %INCLUDE (level 1) file TEMP is file SASHELP.EMEXPL.VARIABLECLUSTERING_MACROS.SOURCE. 51574 +/* Initialize property macro variables */ 51575 +%macro SetProperties; 51576 + %em_checkmacro(name=EM_PROPERTY_MAXCLUS, global=Y, value=DEFAULT); 51577 + %em_checkmacro(name=EM_PROPERTY_HIDEVARIABLE, global=Y, value=Y); 51578 + %em_checkmacro(name=EM_PROPERTY_PRINTOPTION, global=Y, value=SHORT); 51579 + %em_checkmacro(name=EM_PROPERTY_CLUSSOURCE, global=Y, value=CORR); 51580 + %em_checkmacro(name=EM_PROPERTY_CLUSCOMP, global=Y, value=PRINCIPAL); 51581 + %em_checkmacro(name=EM_PROPERTY_CLUSHIERACHY, global=Y, value=Y); 51582 + %em_checkmacro(name=EM_PROPERTY_INCLUDECLASSVAR, global=Y, value=N); 51583 + %em_checkmacro(name=EM_PROPERTY_EXPORTEDCOMP, global=Y, value=CLUSTERCOMP); 51584 + %em_checkmacro(name=EM_PROPERTY_MAXEIGEN, global=Y, value=DEFAULT); 51585 + %em_checkmacro(name=EM_PROPERTY_PROPORTION, global=Y, value=DEFAULT); 51586 + %em_checkmacro(name=EM_PROPERTY_PRINTOPTION, global=Y, value=SHORT); 51587 + %em_checkmacro(name=EM_PROPERTY_TWOSTAGECLUS, global=Y, value=AUTO); 51588 + %em_checkmacro(name=EM_PROPERTY_SUPPRESSSAMPWARN, global=Y, value=N); 51590 +%mend SetProperties; 51592 +%Macro MakeDummyVariables(indata=, 51593 + outvar=, 51594 + outdata=, 51595 + fileref=, 51596 + recreatecmeta=N, /* optional */ 51597 + incmeta=, /* optional */ 51598 + outcmeta=, /* optional */ 51599 + ndummyvars=_ndummyvars 51600 + ); 51601 + %global &ndummyvars; 51603 + proc dmdb batch data=&indata out=_dmdbdat dmdbcat=_dmdbcat classout=_classout;; 51604 + class 51605 + %EM_BINARY_INPUT %EM_NOMINAL_INPUT %EM_ORDINAL_INPUT 51606 + %EM_BINARY_REJECTED %EM_NOMINAL_REJECTED %EM_ORDINAL_REJECTED 51607 + ; 51608 + run; 51609 + %let &ndummyvars = 0; 51610 + data _null_; 51611 + %let dsid = %sysfunc(open(work._classout)); 51612 + %let &ndummyvars = %sysfunc(attrn(&dsid, NOBS)); 51613 + %let dsid = %sysfunc(close(&dsid)); 51614 + run; 51616 + proc dmzip data=_dmdbdat dmdbcat=_dmdbcat; 51617 + input 51618 + %EM_BINARY_INPUT %EM_NOMINAL_INPUT %EM_ORDINAL_INPUT 51619 + %EM_BINARY_REJECTED %EM_NOMINAL_REJECTED %EM_ORDINAL_REJECTED 51620 + / level=nominal stdize=no; 51621 + make outvar = &outvar; 51622 + score data = &indata out =&outdata; 51623 + code file= "&fileref"; 51624 + run; 51625 + %if &recreatecmeta eq Y %then %do; 51626 + proc contents data =&outvar out=_tmpds(keep=NAME LABEL); 51627 + data _tmpds; 51628 + set _tmpds; 51629 + ROLE = 'INPUT'; 51630 + LEVEL = 'INTERVAL'; 51631 + CREATOR='DMZIP'; 51632 + if NAME = '_TYPE_' then delete; 51633 + run; 51634 + data &outcmeta; 51635 + set &incmeta _tmpds; 51636 + run; 51637 + %end; 51638 + proc datasets lib=work nolist; 51639 + delete _dmdbdat _dmdbcat _classout 51640 + %if &recreatecmeta eq Y %then %do; 51641 + _tmpds 51642 + %end; 51643 + ; 51644 + quit; 51645 +%Mend MakeDummyVariables; 51647 +/*--- Determine Optimal Number of Cluster ---- 51648 +%macro FindClusNum(statds=, groupds=, minvariation=); 51649 + %global optnclus; 51650 + data varclus_tmp(drop=_NAME_); 51651 + set &statDs; 51652 + where _type_ ='PROPOR'; 51653 + run; 51654 + proc sort data=varclus_tmp; 51655 + by _NCL_; 51656 + run; 51657 + proc transpose data=varclus_tmp out=varclus_tmp; 51658 + by _NCL_; 51659 + var %EM_INTERVAL_INPUT 51660 + %if &EM_PROPERTY_INCLUDECLASSVAR eq Y %then %do; 51661 + %let dsid = %sysfunc(open(&EM_USER_OUTDUMMY)); 51662 + %let nvar = %sysfunc(attrn(&dsid, NVAR)); 51663 + %do i = 2 %to &nvar; 51664 + %let varname = %sysfunc(varname(&dsid, &i)); 51665 + &varname 51666 + %end; 51667 + %end; 51668 + ; 51669 + run; 51671 + %if &minVariation eq %then %do; 51672 + %let minVariation = &EM_PROPERTY_MINVARIATION; 51673 + %end; 51674 + %if ^(0<&minVariation<100) %then %do; 51675 + %let minVariation = 90; 51676 + %end; 51678 + data _null_; 51679 + set varclus_tmp end=eof; 51680 + by _NCL_; 51681 + retain flag 0; 51682 + if first._ncl_ then flag=0; 51683 + if .<col1 < &minVariation then flag=1; 51684 + if last._ncl_ and ^flag then do; 51685 + call symput('OPTNCL', _ncl_); 51686 + stop; 51687 + end; 51688 + if eof then call symput('OPTNCL', _ncl_); 51689 + run; 51691 + %let optnclus = &OPTNCL; 51693 + data varclus_tmp(drop=_NCL_ _NAME_); 51694 + set &statDs; 51695 + where _type_ in('RSQUARED' 'GROUP') and _NCL_=&OPTNCL; 51696 + run; 51697 + proc sort data=varclus_tmp; 51698 + by _TYPE_; 51699 + run; 51700 + proc transpose data=varclus_tmp out=varclus_tmp; 51701 + by _TYPE_; 51702 + run; 51703 + proc sort data=varclus_tmp; 51704 + by _name_ _type_; 51705 + run; 51707 + proc transpose data=varclus_tmp out=&groupds; 51708 + by _NAME_; 51709 + run; 51710 + proc sort data=&groupDs(rename=(col1=Cluster col2=Rsquare _NAME_=VARIABLE)); 51711 + by Cluster descending Rsquare; 51712 + where Cluster ne 0; 51713 + run; 51714 + proc datasets lib=work nolist mt=(DATA VIEW); 51715 + delete varclus_tmp; 51716 + run; 51717 + quit; 51718 +%mend findClusNum; 51719 +*/ 51721 +%macro getNclusfromTrain(inoutstat=, nc=); 51722 +%global &nc; 51723 +data _null_; 51724 + set &inoutstat end=eof; 51725 + if eof then do; 51726 + call symput("&nc", _ncl_); 51727 + end; 51728 +run; 51729 +%mend getNclusfromTrain; 51731 +%macro MakeDeltaCode(groupds=, outstatscore=, deltacodefile=); 51733 + *--- Build Code to Modify Metadata ---*; 51734 + filename X "&deltacodefile"; 51735 + data _null_; 51736 + FILE X; 51737 + set &groupds end=eof; 51738 + /*by Cluster;*/ 51739 + if _N_=1 then do; 51740 + %if &EM_PROPERTY_INCLUDECLASSVAR eq Y %then %do; 51741 + put "if upcase(strip(ROLE)) ='INPUT' and upcase(strip(LEVEL)) ^='INTERVAL' then ROLE ='REJECTED' ;"; 51742 + %end; 51743 + put "if upcase(strip(ROLE))='INPUT' and upcase(strip(LEVEL))='INTERVAL' then do;"; 51744 + put "if upcase(strip(NAME)) in ("; 51745 + end; 51746 + if Strip(upcase(Selected)) eq 'YES' then do; 51747 + string = '"'!!trim(left(VARIABLE))!!'"'; 51748 + put string; 51749 + end; 51750 + if eof then do; 51751 + put ') then ROLE="INPUT";'; 51752 + put 'else ROLE="REJECTED";'; 51753 + put 'end;'; 51755 + %if %upcase(&EM_PROPERTY_HIDEVARIABLE) eq Y %then %do; 51756 + put 'if upcase(strip(ROLE)) = "REJECTED" then delete ;'; 51757 + %end; 51758 + end; 51759 + run; 51760 + quit; 51762 + filename X; 51764 + quit; 51765 +%mend MakeDeltaCode; 51767 +%macro MakeVarClusCorrData(statds=, corrds=, corrplotds= ); 51768 + %if ^%sysfunc(exist(&statds)) %then %do; 51769 + %goto doendc; 51770 + %end; 51772 + data &corrds(drop=_TYPE_ _NCL_) ; 51773 + set &statds; 51774 + where _type_ eq 'CORR' ; 51775 + run ; 51776 + proc sort data=&corrds; 51777 + by _NAME_ ; 51778 + run ; 51779 + proc transpose data=&corrds out=&corrplotds name=_TMP_; 51780 + BY _NAME_ ; 51781 + run ; 51782 + data &corrplotds; 51783 + length _Y_ $100; 51784 + set &corrplotDs; 51785 + if _LABEL_ ne '' then _Y_=_LABEL_ ; else _Y_=_TMP_ ; 51786 + run ; 51787 + data varclus_match(rename=(_TMP_= _NAME_ _LABEL_=_X_)); 51788 + set &corrplotds; 51789 + where _LABEL_ ne '' ; 51790 + keep _TMP_ _LABEL_ ; 51791 + run ; 51792 + data _null_; 51793 + nobs=0; 51794 + dsid = open('varclus_match'); 51795 + if dsid then do; 51796 + nobs = attrn(dsid, 'NOBS'); 51797 + dsid = close(dsid); 51798 + end; 51799 + call symput ('CORR_NOBS', nobs); 51800 + run; 51801 + %if &corr_nobs %then %do; 51802 + proc sort data=varclus_match; 51803 + by _name_; 51804 + run ; 51805 + proc sort data=&corrplotds; 51806 + by _name_; 51807 + run ; 51808 + data &corrplotds(keep= _X_ _Y_ col1 rename=(col1=Correlation)) ; 51809 + merge varclus_match &corrplotds; 51810 + by _NAME_ ; 51811 + if _X_ eq '' then _X_=_NAME_ ; 51812 + label _X_ = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_variable, noquote))"; 51813 + label _Y_ = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_variable, noquote))"; 51814 + label col1 = "%sysfunc(sasmsg(sashelp.dmine, rpt_correlation_vlabel, noquote))"; 51816 + run ; 51817 + %end; 51818 + %else %do; 51819 + proc sort data=&corrplotds; 51820 + by _name_; 51821 + run ; 51822 + data &corrplotds(keep= _NAME_ _Y_ col1 rename=(_NAME_=_X_ col1=Correlation)) ; 51823 + set &corrplotds; 51824 + label _NAME_ = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_variable, noquote))" ; 51825 + label _Y_ = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_variable, noquote))" ; 51826 + label col1 = "%sysfunc(sasmsg(sashelp.dmine, rpt_correlation_vlabel, noquote))"; 51828 + run ; 51829 + %end; 51830 + proc sort data=&corrplotds; 51831 + by _X_ _Y_; 51832 + run ; 51833 + proc datasets lib=work nolist mt=(DATA VIEW); 51834 + delete varclus_match; 51835 + run; 51836 + quit; 51838 +%doendc: 51840 +%mend MakeVarClusCorrData; 51842 +%macro MakeStatPlotData(statds= , outstatplotds=); 51843 + %if %sysfunc(exist(&statds)) %then %do; 51845 + data varclus_tmp(drop=_NAME_ _NCL_) ; 51846 + set &statDs; 51847 + where _type_ in('MEAN', 'STD', 'N'); 51848 + run ; 51849 + proc transpose data=varclus_tmp out=&outstatplotds; 51850 + id _TYPE_ ; 51851 + run ; 51852 + data &outstatplotds; 51853 + set &outstatplotds(obs=1000); 51854 + label _name_= "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_name, noquote))"; 51855 + label _label_="%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_label, noquote))" ; 51856 + if MEAN ne 0 then SCALEDSTD= STD / MEAN ; 51857 + else SCALEDSTD= STD ; 51858 + label SCALEDSTD = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_scaledstd, noquote))"; 51859 + run ; 51860 + proc sort data=&outstatplotds; 51861 + by descending SCALEDSTD ; 51862 + run ; 51863 + proc datasets lib=work nolist mt=(DATA VIEW); 51864 + delete varclus_tmp; 51865 + run; 51866 + quit; 51867 + %end; 51869 +%mend MakeStatPlotData; 51872 +%macro CreateScoreCode(indata=, ncluscomp=, fileref=); 51873 + %EM_GETNAME(KEY=OUTSTATSCORE, type=DATA); 51874 + data &EM_USER_OUTSTATSCORE; 51875 + set &indata; 51876 + if (_TYPE_ in ('SCORE' 'MEAN' 'STD') and _NCL_ = &ncluscomp ) or (_TYPE_ in ('MEAN' 'STD')); 51877 + if _TYPE_ = 'MEAN' then _NAME_='MEAN'; 51878 + if _TYPE_ = 'STD' then _NAME_='STD'; 51879 + DROP _TYPE_ _NCL_; 51880 + run; 51882 + filename _file_ "&fileRef"; 51884 + data _null_; 51885 + FILE _file_ MOD; 51886 + put ' '; 51887 + put '/*-------------------------------------------------*/'; 51888 + put '/* ' "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_score_title_begin , noquote))" '*/'; 51889 + put '/*-------------------------------------------------*/'; 51890 + put ' '; 51891 + %let dsid = %sysfunc(open(&EM_USER_OUTSTATSCORE)); 51893 + %let nvar = %sysfunc(attrn(&dsid, NVAR)); 51894 + %let vn_name =%sysfunc(varnum(&dsid, _NAME_)); 51896 + %let k = 1; 51897 + %do %while(^%sysfunc(fetch(&dsid))); 51898 + %let _name = %sysfunc(getvarc(&dsid, &vn_name)); 51899 + %if &k > 2 %then %do; 51900 + %let cn = %eval(&k-2); 51901 + put "&_name = 0 ; /*---" "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_score_cluscompnum, noquote, &cn))" "------ */"; 51902 + %end; 51903 + %let k = %eval(&k+1); 51904 + %end; 51906 + %let rc = %sysfunc(rewind(&dsid)); 51908 + %do i= 2 %to &nvar; 51909 + %let _varname = %sysfunc(varname(&dsid, &i)); 51910 + %do %while(^%sysfunc(fetch(&dsid))); 51911 + %let _name = %sysfunc(getvarc(&dsid, &vn_name)); 51912 + %if &_name = MEAN %then 51913 + %let _mean = %sysfunc(getvarn(&dsid, &i)); 51914 + %else %if &_name = STD %then 51915 + %let _std = %sysfunc(getvarn(&dsid, &i)); 51916 + %else %do; 51917 + %let coeff = %sysfunc(getvarn(&dsid, &i)); 51918 + %let abscoeff = %sysfunc(abs(&coeff)); 51919 + %if &abscoeff > 0 %then %do; 51920 + put "&_name = &_name+&coeff * (&_varname - &_mean)/&_std;"; 51921 + %end; 51922 + %end; 51923 + %end; 51924 + %let rc = %sysfunc(rewind(&dsid)); 51925 + %end; 51927 + %let dsid= %sysfunc(close(&dsid)); 51928 + run; 51929 +%mend CreateScoreCode; 51933 +/*---------------------------------------------------------- 51934 + Instead of using %MakeRSquareData, 51935 + %MakeVarClusResultTable at macro2.source is used 51936 + +----------------------------------------------------------*/ 51939 +%macro MakeRSquareData(indata=, inClusRSquare=, outdata=, ncluster=); 51941 +/* modifying from ods rsquare = data */ 51943 +data &outdata(drop= ControlVar NumberOfClusters CurrentCluster); 51944 + Length Cluster $16; 51945 + length Variable $32; 51946 + Length VariableLabel $64; 51947 + set &indata; retain CurrentCluster; 51948 + if NumberOfClusters ^= &ncluster then delete; 51949 + if strip(Cluster) eq '' then Cluster = CurrentCluster; 51950 + CurrentCluster = Cluster; 51951 + run; 51952 +proc sort data =&outdata ; 51953 + by Cluster RsquareRatio; 51954 +run; 51955 +data _tmprsq(drop=index); 51956 + set &outdata; by Cluster; 51957 + if first.Cluster then do; 51958 + index = strip(scan(Cluster,2 )); 51959 + Variable = "Clus"||index; 51960 + VariableLabel = "Cluster Component "||index; 51961 + OwnCluster = 1; 51962 + NextClosest = .; 51963 + RsquareRatio = 0; 51964 + output; 51965 + end; 51966 +run; 51968 +proc sort data = _tmprsq ; 51969 + by Cluster RsquareRatio; 51970 +run; 51971 +data &outdata; 51972 + set &outdata _tmprsq; 51973 +by Cluster; 51974 +run; 51977 +/* Just create the Selected variable with all YES */ 51979 +data &outdata; 51980 + set &outdata; by cluster; 51981 + length Selected $8; 51982 + Selected = 'YES'; 51983 + label OwnCluster = 'R-Sqaure with Cluster Component'; 51984 + label NextClosest = 'R-Sqaure with Next Cluster Component'; 51985 + rename OwnCluster = RSqWithClusterComp; 51986 + rename NextClosest = RSqWithNextClusComp; 51987 +run; 51990 +/* Selected = Y/N will be done %score section ----- 51992 +%if &EM_PROPERTY_EXPORTEDCOMP ne CLUSTERCOMP %then %do; 51993 +data &outdata; 51994 + set &outdata; by cluster; 51995 + length Selected $8; 51996 + if first.Cluster then Selected = 'YES'; 51997 + else Selected = 'NO'; 51998 + label OwnCluster = 'R-Sqaure with Cluster Component'; 51999 + label NextClosest = 'R-Sqaure with Next Cluster Component'; 52000 + rename OwnCluster = RSqWithClusterComp; 52001 + rename NextClosest = RSqWithNextClusComp; 52002 +run; 52003 +%end; 52004 +%else %do; 52005 +data &outdata; 52006 + set &outdata; by cluster; 52007 + if last.Cluster then Selected = 'YES'; 52008 + else Selected = 'NO'; 52009 + label OwnCluster = 'R-Sqaure with Cluster Component'; 52010 + label NextClosest = 'R-Sqaure with Next Cluster Component'; 52011 + rename OwnCluster = RSqWithClusterComp; 52012 + rename NextClosest = RSqWithNextClusComp; 52013 +run; 52014 +%end; 52015 +---------------------------------------------------------------*/ 52017 +%if %sysfunc(exist(&inClusRSquare)) %then %do; 52018 +/* to calculate NextClosestClusRsq */ 52019 +proc transpose data = &inClusRSquare out=_clusRsq; 52020 + by cluster; 52021 + run; 52022 +data _clusRsq; 52023 + set _clusRsq; 52024 + if strip(upcase(Cluster)) eq strip(upcase(_NAME_)) then delete; 52025 +run; 52027 +proc sort data=_clusRsq; 52028 + by cluster col1; 52029 + run; 52030 +data _clusRsq(drop=_NAME_ _LABEL_); 52031 + set _clusRsq; by cluster; 52032 + if last.Cluster then output; 52033 + label COL1 = 'R-Sqaure with Next Cluster Component'; 52034 + rename COL1 = RSqWithNextClusComp; 52035 + rename Cluster = Variable; 52036 + label Cluster = "Variable"; 52037 +run; 52039 +proc sort data =&outdata; 52040 + by Variable; 52041 +run; 52042 +data &outdata; 52043 + merge &outdata _clusRsq; 52044 + by Variable; 52045 +run; 52046 +proc sort data =&outdata; 52047 +by Cluster RsquareRatio; 52048 +run; 52049 +quit; 52050 +%end; 52052 +proc datasets lib = work nolist; 52053 + delete _tmprsq _clusRsq; 52054 + run; 52055 +quit; 52057 +%mend MakeRSquareData; 52060 +/*-------------------------------------------------------------------------*/ 52064 +%macro ModifyCorr(indata=, 52065 + outdata=, 52066 + rsquare = Y 52067 + ); 52068 + data corr_tmp; 52069 + set &indata; 52070 + run; 52071 + proc sql; 52072 + update &indata 52073 + set 52074 + %let dsid = %sysfunc(open(work.corr_tmp)); 52075 + %let nvar = %sysfunc(attrn(&dsid, NVAR)); 52076 + %do i = 4 %to &nvar; 52077 + %let _name = %sysfunc(varname(&dsid, &i)); 52078 + %if &rsquare eq Y %then %let _name_md = &_name.**2; 52079 + %else %let _name_md = &_name; 52080 + %if &i < &nvar %then %do; 52081 + &_name = 1- &_name_md , 52082 + %end; 52083 + %else %do; 52084 + &_name = &_name_md where _TYPE_ contains 'CORR' ; 52085 + %end; 52086 + %end; 52087 + %let dsid= %sysfunc(close(&dsid)); 52089 + select * from &indata; 52090 + run; 52091 + proc datasets lib = work nolist; 52092 + delete corr_tmp; 52093 + run; 52094 + quit; 52096 + %mend ModifyCorr; 52098 + %macro MakeClusStructCorrData(indata=,outdata=, ncluster=, Rsquare=N); 52099 + data &outdata(drop= _NCL_ _TYPE_); 52100 + set &indata; 52101 + if ^(strip(_NCL_) eq &ncluster and strip(_TYPE_) eq 'STRUCTUR') then delete; 52102 + rename _NAME_ = Cluster; 52103 + label _NAME_ = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_clustername, noquote))"; 52104 + run; 52105 + %if &RSquare eq Y %then %do; 52106 + data corr_tmp; 52107 + set &outdata; 52108 + run; 52110 + data &outdata(drop=i); 52111 + set &outdata; 52112 + %let dsid = %sysfunc(open(work.corr_tmp)); 52113 + %let nvar = %sysfunc(attrn(&dsid, NVAR)); 52114 + %do i = 2 %to &nvar; 52115 + %let _name = %sysfunc(varname(&dsid, &i)); 52116 + %let _name_md = &_name.**2; 52117 + &_name = &_name_md; 52118 + %end; 52119 + %let dsid= %sysfunc(close(&dsid)); 52120 + run; 52121 + proc datasets lib = work nolist; 52122 + delete corr_tmp; 52123 + run; 52125 + %end; 52126 + quit; 52127 +%mend MakeClusStructCorrData; 52129 +%macro MakeInterClusCorrData(indata=, outdata=, ncluster=, RSquare=N, makeplotds=N, plotds=); 52130 + data &outdata(drop= _NCL_ _TYPE_); 52131 + set &indata; 52132 + if ^(strip(_NCL_) eq &ncluster and strip(_TYPE_) eq 'CCORR') then delete; 52133 + rename _NAME_ = Cluster; 52134 + label _NAME_ = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_clustername, noquote))"; 52135 + run; 52136 + data corr_tmp; 52137 + set &outdata; 52138 + run; 52140 + %let dsid = %sysfunc(open(work.corr_tmp)); 52141 + %let nclus2= %eval(&ncluster+1); 52142 + data &outdata; 52143 + set &outdata; 52144 + %do i = 2 %to &nclus2; 52145 + %let i_1 = %eval(&i-1); 52146 + %let _name = %sysfunc(varname(&dsid, &i)); 52147 + %let _newName = Clus&i_1; 52148 + rename &_name = &_newName; ; 52149 + *label &_name ="Cluster &i_1"; 52150 + label &_name = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_clusternum, noquote, &i_1))"; 52151 + %end; 52152 + keep Cluster 52153 + %do i = 2 %to &nclus2; 52154 + %let _name = %sysfunc(varname(&dsid, &i)); 52155 + &_name 52156 + %end; 52157 + ; 52158 + %let dsid= %sysfunc(close(&dsid)); 52159 + run; 52160 + quit; 52162 + %if &RSquare eq Y %then %do; 52164 + data corr_tmp; 52165 + set &outdata; 52166 + run; 52168 + data &outdata(drop=i); 52169 + set &outdata; 52170 + %let dsid = %sysfunc(open(work.corr_tmp)); 52171 + %let nvar = %sysfunc(attrn(&dsid, NVAR)); 52172 + %do i = 2 %to &nvar; 52173 + %let _name = %sysfunc(varname(&dsid, &i)); 52174 + %let _name_md = &_name.**2; 52175 + &_name = &_name_md; 52176 + %end; 52177 + %let dsid= %sysfunc(close(&dsid)); 52178 + run; 52179 + %end; 52181 + %if &makeplotds eq Y %then %do; 52182 + proc transpose data = &outdata 52183 + out=&plotds(drop=_LABEL_ rename=(_NAME_ = Y Cluster=X Col1= Correlation)); 52184 + by cluster; 52185 + run; 52186 + data &plotds; 52187 + set &plotds; 52188 + label x="%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_cluster, noquote))"; 52189 + label Y="%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_cluster, noquote))"; 52190 + run; 52191 + %end; 52192 + proc datasets lib = work nolist; 52193 + delete corr_tmp; 52194 + run; 52195 + quit; 52196 +%mend MakeInterClusCorrData; 52199 +%macro MakeClusConstellData(indata=, outlink=, outnode=); 52201 +data &outlink(drop = Selected); 52202 + set &indata; 52203 + LINKID = _N_; 52204 + label LINKID = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_linkid, noquote))"; 52205 + if strip(upcase(Cluster)) eq strip(upcase(Variable)) then Variable = ClosestCluster; 52206 +run; 52207 +data &outnode(keep=NODEID TYPE LABEL); 52208 + set &indata; 52209 + length TYPE $16; 52210 + rename Variable = NODEID; 52211 + label Variable= "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_nodeidvar, noquote))"; 52212 + if strip(upcase(Cluster)) eq strip(upcase(Variable)) 52213 + then TYPE = "CLUSTER"; 52214 + else TYPE="VARIABLE"; 52215 + label TYPE = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_nodetype, noquote))"; 52216 +run; 52217 +quit; 52218 +%mend MakeClusConstellData; 52222 +%macro MakeClusConstellData(indata=, outlink=, outnode=); 52224 +data &outlink(drop = Selected); 52225 + set &indata; 52226 + LINKID = _N_; 52227 + label LINKID = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_linkid, noquote))"; 52228 + if strip(upcase(Cluster)) eq strip(upcase(Variable)) then Variable = ClosestCluster; 52229 +run; 52230 +data &outnode(keep=NODEID TYPE LABEL); 52231 + set &indata; 52232 + length TYPE $16; 52233 + rename Variable = NODEID; 52234 + label Variable= "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_nodeidvar, noquote))"; 52235 + if strip(upcase(Cluster)) eq strip(upcase(Variable)) 52236 + then TYPE = "CLUSTER"; 52237 + else TYPE="VARIABLE"; 52238 + label TYPE = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_nodetype, noquote))"; 52239 +run; 52240 +quit; 52241 +%mend MakeClusConstellData; 52244 +/*--- This will work only when inds is not a view data ------- 52246 +%macro getNVarNObs(inds=, nvar=, nobs=); 52248 + %global &nvar; 52249 + %global &nobs; 52250 + data _null_; 52251 + dsid = open("&inds"); 52252 + nv = attrn(dsid, 'NVAR'); 52253 + no = attrn(dsid, 'NOBS'); 52254 + dsid = close(dsid); 52255 + call symput("&nvar", nv); 52256 + call symput("&nobs", no); 52257 + run; 52258 + quit; 52259 +%mend getNVarNObs; 52261 ++---------------------------------------------------------------*/ 52264 +%macro getNVar(inds=, nvar=); 52265 + %global &nvar; 52266 + data _null_; 52267 + dsid = open("&inds"); 52268 + nv = attrn(dsid, 'NVAR'); 52269 + dsid = close(dsid); 52270 + call symput("&nvar", nv); 52271 + run; 52272 + quit; 52273 +%mend getNVar; 52277 +%macro getNObs(inds=, nobs=); 52278 + %global &nobs; 52279 + data _null_; 52280 + set &inds end=eof; 52281 + if eof then call symput("&nobs", _N_); 52282 + run; 52283 + quit; 52284 +%mend getNObs; 52286 +%Macro CreateVarclusMeta(trainnum=); 52287 + %EM_GETNAME(KEY=VARCLUSMETA, TYPE=DATA); 52288 + data &EM_USER_VARCLUSMETA; 52289 + length TrainNum 8.; 52290 + length NewTrain $8; 52291 + length NGCluster 8.; 52292 + length ExportedComp $16; 52293 + length HideVariable $8; 52294 + TrainNum = &trainnum; 52295 + NewTrain = "Y"; 52296 + ExportedComp = "&EM_PROPERTY_EXPORTEDCOMP"; 52297 + HideVariable = "&EM_PROPERTY_HIDEVARIABLE"; 52298 + NGCluster = 0; /* zero means no twostage */ 52299 + run; 52300 + quit; 52301 +%mend CreateVarclusMeta; NOTE: %INCLUDE (level 1) ending. NOTE: %INCLUDE (level 1) file TEMP is file SASHELP.EMEXPL.VARIABLECLUSTERING_MACROS2.SOURCE. 52303 +%macro MakeInterClusCorrData(indata=, outdata=, ncluster=, globalclusid=, RSquare=N, makeplotds=N, plotds=); 52304 + data &outdata(drop= _NCL_ _TYPE_); 52305 + set &indata; 52306 + if ^(strip(_NCL_) eq &ncluster and strip(_TYPE_) eq 'CCORR') then delete; 52307 + %if &globalclusid ne %then %do; 52308 + _NAME_ = "GC&globalclusid._"||upcase(_NAME_); 52309 + rename _NAME_ = Cluster; 52310 + %end; 52311 + %else %do; 52312 + _NAME_ = upcase(_NAME_); 52313 + rename _NAME_ = Cluster; 52314 + %end; 52315 + label _NAME_ = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_clustername, noquote))"; 52316 + run; 52317 + data corr_tmp; 52318 + set &outdata; 52319 + run; 52321 + %let dsid = %sysfunc(open(work.corr_tmp)); 52322 + %let nclus2= %eval(&ncluster+1); 52323 + data &outdata; 52324 + set &outdata; 52325 + %do i = 2 %to &nclus2; 52326 + %let i_1 = %eval(&i-1); 52327 + %let _name = %sysfunc(varname(&dsid, &i)); 52328 + %if &globalclusid ne %then 52329 + %do; %let _newName = GC&globalclusid._CLUS&i_1; 52330 + rename &_name = &_newName; 52331 + *label &_name ="GC &globalclusid : Cluster &i_1"; 52332 + label &_name = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_gc_clusternum, noquote, &globalclusid, &i_1))"; 52333 + %end; 52334 + %else 52335 + %do; %let _newName = CLUS&i_1; 52336 + rename &_name = &_newName; 52337 + *label &_name ="Cluster &i_1"; 52338 + label &_name = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_clusternum, noquote, &i_1))"; 52339 + %end; 52340 + %end; 52341 + keep Cluster 52342 + %do i = 2 %to &nclus2; 52343 + %let _name = %sysfunc(varname(&dsid, &i)); 52344 + &_name 52345 + %end; 52346 + ; 52347 + %let dsid= %sysfunc(close(&dsid)); 52348 + run; 52349 + quit; 52351 + %if &RSquare eq Y %then %do; 52353 + data corr_tmp; 52354 + set &outdata; 52355 + run; 52357 + data &outdata; 52358 + set &outdata; 52359 + %let dsid = %sysfunc(open(work.corr_tmp)); 52360 + %let nvar = %sysfunc(attrn(&dsid, NVAR)); 52361 + %do i = 2 %to &nvar; 52362 + %let _name = %sysfunc(varname(&dsid, &i)); 52363 + %let _name_md = &_name.**2; 52364 + &_name = &_name_md; 52365 + %end; 52366 + %let dsid= %sysfunc(close(&dsid)); 52367 + run; 52368 + %end; 52370 + %if &makeplotds eq Y %then %do; 52371 + proc transpose data = &outdata 52372 + out=&plotds(drop=_LABEL_ rename=(_NAME_ = Y Cluster=X Col1= Correlation)); 52373 + by cluster; 52374 + run; 52375 + data &plotds; 52376 + set &plotds; 52377 + label x="%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_cluster, noquote))"; 52378 + label Y="%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_cluster, noquote))"; 52379 + label Correlation="%sysfunc(sasmsg(sashelp.dmine, rpt_correlation_vlabel, noquote))"; 52380 + run; 52381 + %end; 52382 + proc datasets lib = work nolist; 52383 + delete corr_tmp; 52384 + run; 52385 + quit; 52386 +%mend MakeInterClusCorrData; 52388 +%macro MakeOwnRSquare(indata=, outdata=, ncluster=, globalclusid=); 52389 + data _tmpds(drop= _NCL_); 52390 + set &indata; 52391 + if ^(strip(_NCL_) eq &ncluster and strip(_TYPE_) in ('GROUP','RSQUARED')) then delete; 52392 + %if &globalclusid ne %then %do; 52393 + _NAME_ = "GC&globalclusid."; 52394 + rename _NAME_ = Cluster; 52395 + %end; 52396 + %else %do; 52397 + _NAME_ = "CLUS"; 52398 + rename _NAME_ = Cluster; 52399 + %end; 52400 + label _NAME_ = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_clustername, noquote))"; 52401 + run; 52402 + proc transpose data =_tmpds out =&outdata; 52403 + run; 52405 + data &outdata(drop=COL1); 52406 + %if &globalclusid ne %then %do; 52407 + length GCluster $16; 52408 + %end; 52409 + length Cluster $32; 52410 + length _NAME_ $32; 52411 + set &outdata; 52412 + _NAME_ = upcase(_NAME_); 52413 + rename _NAME_=Variable; 52414 + *label _NAME_="Variable"; 52415 + label _NAME_ = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_variable, noquote))" ; 52416 + label Cluster = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_cluster, noquote))" ; 52417 + label GCluster = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_gcluster, noquote))" ; 52419 + %if &globalclusid ne %then %do; 52420 + GCluster = "GC&globalclusid"; 52421 + Cluster = "GC&globalclusid._CLUS"||strip(COL1); 52422 + %end; 52423 + %else %do; 52424 + Cluster = "CLUS"||strip(COL1); 52425 + %end; 52426 + rename COL2 = RSqWithOwnClusComp; 52427 + *label COL2 = "R-Square With Own Cluster Component"; 52428 + label COL2 = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_ownrsq, noquote))" ; 52430 + run; 52431 + proc sort data =&outdata; 52432 + by Cluster RSqWithOwnClusComp; 52433 + run; 52434 + proc datasets lib = work nolist; 52435 + delete _tmpds; 52436 + run; 52437 + quit; 52438 +%mend MakeOwnRSquare; 52440 +%macro MakeClusStructCorrData(indata=, outdata=, globalclusid=, ncluster=, Rsquare=N); 52441 + data &outdata(drop= _NCL_ _TYPE_); 52442 + %if &globalclusid ne %then %do; 52443 + length GCluster $16; 52444 + %end; 52445 + set &indata; 52446 + if ^(strip(_NCL_) eq &ncluster and strip(_TYPE_) eq 'STRUCTUR') then delete; 52447 + %if &globalclusid ne %then %do; 52448 + GCluster = "GC&globalclusid"; 52449 + _NAME_ = "GC&globalclusid._"||upcase(_NAME_); 52450 + rename _NAME_ = Cluster; 52451 + label _NAME_ = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_cluster, noquote))" ; 52452 + label GCluster = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_gcluster, noquote))" ; 52454 + %end; 52455 + %else %do; 52456 + _NAME_ = upcase(_NAME_); 52457 + rename _NAME_ = Cluster; 52458 + label _NAME_ = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_cluster, noquote))" ; 52459 + label GCluster = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_gcluster, noquote))" ; 52461 + %end; 52462 + run; 52463 + %if &RSquare eq Y %then %do; 52464 + data corr_tmp; 52465 + set &outdata; 52466 + run; 52467 + %let istart = 2; 52468 + %if &globalclusid ne %then %let istart = 3; 52469 + data &outdata; 52470 + set &outdata; 52471 + %let dsid = %sysfunc(open(work.corr_tmp)); 52472 + %let nvar = %sysfunc(attrn(&dsid, NVAR)); 52473 + %do i =&istart %to &nvar; 52474 + %let _name = %sysfunc(varname(&dsid, &i)); 52475 + %let _name_md = &_name.**2; 52476 + &_name = &_name_md; 52477 + %end; 52478 + %let dsid= %sysfunc(close(&dsid)); 52479 + run; 52480 + proc datasets lib = work nolist; 52481 + delete corr_tmp; 52482 + run; 52483 + %end; 52484 + quit; 52485 +%mend MakeClusStructCorrData; 52487 +/* 52488 +%MakeClusStructCorrData(indata=playpen._outstat, outdata=_structrsq , ncluster=7, Rsquare=Y); 52489 +*/ 52491 +%macro FindNextClosestClusByVar(indata=, outdata=, globalclusid=, ncluster=); 52493 + /* The indata should be the outdata 52494 + from %MakeClusStructCorrData(indata=, outdata=, ); */ 52496 + proc sort data =&indata out=_tmpclusRsq; 52497 + by cluster; 52498 + run; 52500 + proc transpose data =_tmpclusRsq out=_tmpclusRsq; 52501 + by cluster; 52502 + run; 52504 + proc sort data=_tmpclusRsq; 52505 + by _NAME_ COL1; 52506 + run; 52508 + data _tmpclusRsq; 52509 + length _NAME_ $32; 52510 + set _tmpclusRsq; by _NAME_; 52511 + _NAME_ = upcase(_NAME_); 52512 + %if &ncluster ne 1 %then %do; 52513 + if last._NAME_ then delete; 52514 + %end; 52515 + %else %do; 52516 + COL1 = 0; 52517 + %end; 52518 + run; 52519 + /* need to sort again */ 52520 + proc sort data=_tmpclusRsq; 52521 + by _NAME_ COL1; 52522 + run; 52524 + data &outdata; 52525 + set _tmpclusRsq; by _NAME_; 52526 + Cluster = upcase(Cluster); 52527 + if last._NAME_ then output; 52528 + *label COL1 = 'R-Sqaure with Next Cluster Component'; 52529 + label COL1 = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_nextrsq, noquote))" ; 52530 + rename COL1 = RSqWithNextClusComp; 52531 + Cluster = upcase(Cluster); 52532 + rename Cluster = ClosestCluster; 52533 + *label Cluster = "Next Closest Cluster"; 52534 + label Cluster = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_nextclus, noquote))" ; 52535 + rename _NAME_ = Variable; 52536 + label _NAME_ = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_variable, noquote))" ; 52537 + run; 52540 + %if &globalclusid ne %then %do; 52541 + data &outdata; 52542 + length GCluster $16; 52543 + set &outdata; 52544 + GCluster = "GC&globalclusid"; 52545 + run; 52546 + %end; 52547 + proc datasets lib = work nolist; 52548 + delete _tmpclusRsq; 52549 + run; 52550 + quit; 52551 +%mend FindNextClosestClusByVar; 52554 +%macro FindNextClosestClusByCluster(indata=, outdata=, globalclusid=, ncluster=); 52555 + /* The indata should be the outdata from %MakeInterClusCorrData(indata=, outdata=, ); */ 52556 + proc sort data =&indata out=_tmpclusRsq; 52557 + by cluster; 52558 + run; 52559 + proc transpose data =_tmpclusRsq out=_tmpclusRsq; 52560 + by cluster; 52561 + run; 52562 + proc sort data=_tmpclusRsq; 52563 + by _NAME_ col1; 52564 + run; 52565 + data _tmpclusRsq; 52566 + length _NAME_ $32; 52567 + set _tmpclusRsq; by _NAME_; 52568 + _NAME_ = upcase(_NAME_); 52569 + %if &ncluster ne 1 %then %do; 52570 + if last._NAME_ then delete; 52571 + %end; 52572 + %else %do; 52573 + COL1 = 0; 52574 + %end; 52575 + run; 52576 + data &outdata; 52577 + set _tmpclusRsq; by _NAME_; 52578 + Cluster = upcase(Cluster); 52579 + if last._NAME_ then output; 52580 + *label COL1 = 'R-Sqaure with Next Cluster Component'; 52581 + label COL1 = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_nextrsq, noquote))" ; 52582 + rename COL1 = RSqWithNextClusComp; 52583 + Cluster = upcase(Cluster); 52584 + rename Cluster = ClosestCluster; 52585 + *label Cluster = "Next Closest Cluster"; 52586 + label Cluster = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_nextclus, noquote))" ; 52587 + rename _NAME_ = Variable; 52588 + *label _NAME_ = "Variable"; 52589 + label _NAME_ = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_variable, noquote))" ; 52591 + run; 52592 + %if &globalclusid ne %then %do; 52593 + data &outdata; 52594 + length GCluster $16; 52595 + set &outdata; 52596 + GCluster = "GC&globalclusid"; 52597 + run; 52598 + %end; 52600 + proc datasets lib = work nolist; 52601 + delete _tmpclusRsq; 52602 + run; 52604 + quit; 52605 +%mend FindNextClosestClusByCluster; 52607 +%macro MakeVarClusResultTable(indata1=, indata2=, indata3=, outdata=, globalclusid=, ncluster=, selectedcomp=clustercomp); 52608 +/*---- 52609 + indata1=_ownRsq, indata2=_nextVarRsq, indata3=_nextClusRSq, 52610 +-----------*/ 52612 +proc sort data =&indata1; 52613 + by Variable; 52614 +run; 52615 +proc sort data =&indata2; 52616 + by Variable; 52617 +run; 52618 +data &outdata; 52619 + merge &indata1 &indata2; 52620 + by Variable; 52621 + length Type $16; 52622 + Type = 'Variable'; 52623 + *label Type ='Type'; 52624 + label Type = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_type, noquote))" ; 52625 +run; 52628 +data &indata3; 52629 + set &indata3 ; 52630 + length RSqWithOwnClusComp 8.; 52631 + Cluster = Variable; 52632 + RSqWithOwnClusComp = 1; 52633 + *label RSqWithOwnClusComp = "R-Square With Own Cluster Component"; 52634 + label RSqWithOwnClusComp = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_ownrsq, noquote))" ; 52635 + length Type $16; 52636 + Type = 'ClusterComp'; 52637 + label Type = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_type, noquote))" ; 52639 +; 52640 +run; 52642 +proc sort data=&outdata; 52643 + by Cluster; 52644 +run; 52645 +proc sort data =&indata3; 52646 + by Cluster; 52647 +run; 52649 +data &outdata; 52650 + set &outdata &indata3; 52651 + by Cluster; 52652 +run; 52655 +/* Create the Selected variable with all YES */ 52657 +data &outdata; 52658 + set &outdata; 52659 + length RsqRatio 8.; 52660 + length Selected $8; 52661 + *label RSqRatio = "1-R**2 Ratio"; 52662 + label RSqRatio = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_oneminusrsq, noquote))" ; 52663 + *label Selected = "Variable Selected"; 52664 + label Selected = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_varselected, noquote))" ; 52665 + RsqRatio = (1-RSqWithOwnClusComp)/(1-RSqWithNextClusComp); 52666 + Selected ='YES'; 52667 + rename _LABEL_ = Label; 52668 + label _LABEL_ = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_label, noquote))" ; 52669 +run; 52672 +/*---- Selected = Y/N will be assigened at the %score -------+ 52673 + Just create the Selected variable with all YES at the step above 52674 + +------------------------------------------------------------------------------+ 52676 +proc sort data=&outdata; 52677 + by Cluster RsqRatio; 52678 +run; 52680 +%if &selectedcomp eq CLUSTERCOMP %then %do; 52681 +data &outdata; 52682 + set &outdata; by Cluster; 52683 + length Selected $8; 52684 + label Selected = "Variable Selected"; 52685 + if first.Cluster then Selected ='Yes'; 52686 + else Selected = 'No'; 52687 + run; 52688 +%end; 52689 +%else %do; 52690 +data &outdata(drop = _var _varchange); 52691 + set &outdata; retain _var 0; by Cluster; 52692 + length Selected $8; 52693 + label Selected = "Variable Selected"; 52694 + if first.Cluster then _varchange = 0; 52695 + else _varchange =1; 52696 + if _var ne _varchange then Selected = 'Yes'; 52697 + else Selected = 'No'; 52698 + if last.cluster then _var = 0; 52699 + else _var = _varchange; 52700 +run; 52701 +%end; 52703 +--------------------------------------------------------------*/ 52705 +quit; 52706 +%mend MakeVarClusResultTable; 52708 +%Macro MakePlotDataFromCorrTable(indata=, outdata=, globalclusid=); 52709 + proc sort data =&indata; 52710 + by cluster; 52711 + run; 52712 + proc transpose data =&indata 52713 + out=&outdata(drop=_LABEL_ rename=(_NAME_ = Y Cluster=X Col1= Correlation)); 52714 + by cluster; 52715 + run; 52716 + data &outdata; 52717 + set &outdata; 52718 + label x= "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_cluster, noquote))" ; 52719 + label Y= "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_cluster, noquote))" ; 52720 + label Correlation = "%sysfunc(sasmsg(sashelp.dmine, rpt_correlation_vlabel, noquote))" ; 52721 + run; 52722 + %if &globalclusid ne %then %do; 52723 + data &outdata; 52724 + Length GCluster $16; 52725 + label GCluster = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_gcluster, noquote))" ; 52726 + set &outdata; 52727 + GCluster ="GC&globalclusid."; 52728 + run; 52729 + %end; 52731 +%Mend MakePlotDataFromCorrTable; 52734 +%macro MakeCorrelation(indata=, 52735 + outstat=_tmpoutstat, 52736 + corrmatrix=N, 52737 + outcorr=_tmpoutcorr, 52738 + includeclassvar=N, 52739 + target=, 52740 + freq=, 52741 + weight=); 52742 + %if &target eq %then %do; 52744 + proc varclus data=&indata outstat=&outstat hi maxclusters=1 noprint; 52745 + var %EM_INTERVAL_INPUT %EM_INTERVAL_REJECTED 52746 + %if &includeclassvar eq Y %then %do; 52747 + %let dsid = %sysfunc(open(&EM_USER_OUTDUMMY)); 52748 + %let nvar = %sysfunc(attrn(&dsid, NVAR)); 52749 + %do i = 2 %to &nvar; 52750 + %let varname = %sysfunc(varname(&dsid, &i)); 52751 + &varname 52752 + %end; 52753 + %end; 52754 + ; 52755 + %if &freq ne %then %do; 52756 + freq &freq; 52757 + %end; 52758 + %if &weight ne %then %do; 52759 + weight &weight; 52760 + %end; 52762 + run; 52763 + %if &corrmatrix eq Y %then %do; 52764 + data &outcorr (drop = _NCL_ _TYPE_); 52765 + set &outstat ; 52766 + if _TYPE_ ='CORR' then output; 52767 + run; 52768 + %end; 52769 + %end; 52770 + %else %do; 52771 + proc corr data=&indata outp=&outstat noprint; 52772 + var 52773 + %let dsid = %sysfunc(open(&indata)); 52774 + %let nvar = %sysfunc(attrn(&dsid, NVAR)); 52775 + %do i = 1 %to &nvar; 52776 + %let _name = %sysfunc(varname(&dsid, &i)); 52777 + %if &_name ne &target %then; 52778 + &_name 52779 + %end; 52780 + %let dsid= %sysfunc(close(&dsid)); 52781 + ; 52782 + with &target; 52783 + run; 52784 + %end; 52785 + quit; 52786 +%mend MakeCorrelation; 52789 +%macro MakeCorrelationDistance(indata=, 52790 + outdata=, 52791 + rsquare = N 52792 + ); 52793 + data corr_tmp; 52794 + set &indata; 52795 + if _N_ = 1 then do; 52796 + output; 52797 + stop; 52798 + end; 52799 + run; 52800 + %if &outdata ne %then %let _outdata = &outdata; 52801 + %else %let _outdata = &indata; 52803 + data &_outdata; 52804 + set &indata; 52806 + %let dsid = %sysfunc(open(work.corr_tmp)); 52807 + %let nvar = %sysfunc(attrn(&dsid, NVAR)); 52808 + %do i = 2 %to &nvar; 52809 + %let _name = %sysfunc(varname(&dsid, &i)); 52810 + %if &rsquare eq Y %then %let _name_md = &_name.**2; 52811 + %else %let _name_md = &_name; 52812 + &_name = 1- &_name_md; 52813 + %end; 52814 + %let dsid= %sysfunc(close(&dsid)); 52815 + run; 52816 + proc datasets lib = work nolist; 52817 + delete corr_tmp; 52818 + run; 52819 + quit; 52820 + %mend MakeCorrelationDistance; 52823 +%macro UpdateOutStatCorrToDistance(indata=, /* indata should be a outstat from proc varclus */ 52824 + rsquare = N 52825 + ); 52826 + data corr_tmp; 52827 + set &indata; 52828 + run; 52829 + proc sql noprint; 52830 + update &indata 52831 + set 52832 + %let dsid = %sysfunc(open(work.corr_tmp)); 52833 + %let nvar = %sysfunc(attrn(&dsid, NVAR)); 52834 + %do i = 4 %to &nvar; 52835 + %let _name = %sysfunc(varname(&dsid, &i)); 52836 + %if &rsquare eq Y %then %let _name_md = &_name.**2; 52837 + %else %let _name_md = &_name; 52838 + %if &i < &nvar %then %do; 52839 + &_name = 1- &_name_md , 52840 + %end; 52841 + %else %do; 52842 + &_name = &_name_md where _TYPE_ eq 'CORR' ; 52843 + %end; 52844 + %end; 52845 + %let dsid= %sysfunc(close(&dsid)); 52847 + select * from &indata; 52848 + run; 52849 + data &indata( drop = _NCL_); 52850 + set &indata; 52851 + if _TYPE_ not in ('CORR', 'STD', 'N', 'MEAN') then delete; 52852 + if _TYPE_ ='CORR' then _TYPE_ ='DISTANCE'; 52853 + run; 52854 + data &indata(DROP = _NCL_); 52855 + set &indata; 52856 + if _TYPE_ = 'CORR' then _TYPE_ ='DISTANCE'; 52857 + if _TYPE_ not in ('DISTANCE', 'N', 'STD', 'MEAN') then delete; 52858 + rename _NAME_ = _VAR_; 52859 + run; 52860 + proc datasets lib = work nolist; 52861 + delete corr_tmp; 52862 + run; 52863 + quit; 52864 + %mend UpdateOutStatCorrToDistance; 52867 +%macro HierClusWithCorr(indata= , 52868 + ncluster=, 52869 + method = Ward, 52870 + outtree = _outtree, 52871 + idvar =_VAR_, 52872 + outdata=, 52873 + rescore = N, 52874 + newncluster= 52875 + ); 52876 + %global &newncluster; 52877 + %if &rescore ne Y %then %do; 52878 + proc cluster data=&indata(type=Distance where=(upcase(strip(_TYPE_)) = "DISTANCE")) 52879 + method=&method outtree=&outtree noprint; 52880 + id &idvar; 52881 + run; 52882 + %end; 52883 + proc tree data=&outtree nclusters = &ncluster out=&outdata noprint; 52884 + run; 52885 + /* ----- Check some variables like CL1, CL5..., remove them ----*/ 52886 + proc contents data =&indata out=_outcontent(keep=NAME) noprint; 52887 + run; 52888 + data _outcontent; 52889 + set _outcontent; 52890 + if NAME in ('_TYPE_' , '_VAR_') then delete; 52891 + index = 1; 52892 + rename NAME = _NAME_; 52893 + run; 52894 + proc sort data=_outcontent; 52895 + by _NAME_; 52896 + run; 52897 + proc sort data =&outdata; 52898 + by _NAME_; 52899 + run; 52900 + data &outdata(drop=index); 52901 + merge &outdata _outcontent; 52902 + by _NAME_; 52903 + if index = . then delete; 52904 + run; 52905 + /*-----------------------------------------------------------*/ 52906 + data &outdata; 52907 + length CLUSNAME $16; 52908 + set &outdata; 52909 + if CLUSTER > &ncluster then delete; 52910 + CLUSNAME='GC'||strip(CLUSTER); 52911 + *label CLUSNAME = "Cluster Name"; 52912 + label CLUSNAME = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_clustername, noquote))" ; 52913 + rename _NAME_ = VARIABLE ; 52914 + *label _NAME_ = "Variable"; 52915 + *label CLUSTER = "Cluster"; 52916 + label _NAME_ ="%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_variable, noquote))" ; 52917 + label CLUSTER ="%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_cluster, noquote))" ; 52918 + run; 52919 + proc sort data=&outdata out=&outdata; 52920 + by CLUSTER; 52921 + run; 52922 + proc means data =&outdata noprint; 52923 + output out=_meanout; 52924 + run; 52925 + data _null_; 52926 + set _meanout; 52927 + if strip(_STAT_) eq 'MAX' then do; 52928 + call symput("&newncluster", CLUSTER); 52929 + stop; 52930 + end; 52931 + run; 52933 + proc datasets lib = work nolist; 52934 + delete _outcontent _meanout; 52935 + run; 52936 + quit; 52937 +%mend HierClusWithCorr; 52939 +%macro CreateScoreCode2(indata=, ncluscomp=, globalclusid=, fileref=); 52941 + data _tmpindata; 52942 + set &indata; 52943 + if (_TYPE_ in ('SCORE' 'MEAN' 'STD') and _NCL_ = &ncluscomp ) or (_TYPE_ in ('MEAN' 'STD')); 52944 + if _TYPE_ = 'MEAN' then _NAME_='MEAN'; 52945 + if _TYPE_ = 'STD' then _NAME_='STD'; 52946 + if _TYPE_ = 'SCORE' then _NAME_=upcase("GC&globalclusid._"||_NAME_); 52947 + DROP _TYPE_ _NCL_; 52948 + run; 52950 + filename _file_ "&fileRef"; 52952 + data _null_; 52953 + FILE _file_ MOD; 52954 + put ' '; 52955 + put "/*-------------------------------------------------*/"; 52956 + put '/* ' "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_score_title_gclus, noquote, &globalclusid))" '*/'; 52957 + put "/*-------------------------------------------------*/"; 52958 + put ' '; 52959 + %let dsid = %sysfunc(open(work._tmpindata)); 52961 + %let nvar = %sysfunc(attrn(&dsid, NVAR)); 52962 + %let vn_name =%sysfunc(varnum(&dsid, _NAME_)); 52963 + %let k = 1; 52964 + %do %while(^%sysfunc(fetch(&dsid))); 52965 + %let _name = %sysfunc(getvarc(&dsid, &vn_name)); 52966 + %if &k > 2 %then %do; 52967 + %let cn = %eval(&k-2); 52968 + put "&_name = 0 ; /*---" "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_score_gcluscompnum, noquote, &globalclusid, &cn))" "------ */"; 52969 + %end; 52970 + %let k = %eval(&k+1); 52972 + %end; 52973 + %let rc = %sysfunc(rewind(&dsid)); 52975 + %do j= 2 %to &nvar; 52976 + %let _varname = %sysfunc(varname(&dsid, &j)); 52977 + %do %while(^%sysfunc(fetch(&dsid))); 52978 + %let _name = %sysfunc(getvarc(&dsid, &vn_name)); 52979 + %if &_name = MEAN %then 52980 + %let _mean = %sysfunc(getvarn(&dsid, &j)); 52981 + %else %if &_name = STD %then 52982 + %let _std = %sysfunc(getvarn(&dsid, &j)); 52983 + %else %do; 52984 + %let coeff = %sysfunc(getvarn(&dsid, &j)); 52985 + %let abscoeff = %sysfunc(abs(&coeff)); 52986 + %if &abscoeff > 0 %then %do; 52987 + put "&_name = &_name+&coeff * (&_varname - &_mean)/&_std;"; 52988 + %end; 52989 + %end; 52990 + %end; 52991 + %let rc = %sysfunc(rewind(&dsid)); 52992 + %end; 52994 + %let dsid= %sysfunc(close(&dsid)); 52996 + run; 52998 + filename _file_; 52999 + proc datasets lib = work nolist; 53000 + delete _tmpindata; 53001 + run; 53002 + quit; 53003 +%mend CreateScoreCode2; 53006 +%macro MakeDeltaCode2(groupds=,deltacodefile=); 53008 + /*--- Build Code to Modify Metadata ---*/ 53009 + filename X "&deltacodefile"; 53010 + data _null_; 53011 + FILE X; 53012 + set &groupds end=eof; 53013 + if _N_=1 then do; 53014 + %if &EM_PROPERTY_INCLUDECLASSVAR eq Y %then %do; 53015 + put "if upcase(strip(ROLE)) ='INPUT' and upcase(strip(LEVEL)) ^='INTERVAL' then ROLE ='REJECTED' ;"; 53016 + %end; 53017 + put "if upcase(strip(ROLE))='INPUT' and upcase(strip(LEVEL))='INTERVAL' then do;"; 53018 + put "if upcase(strip(NAME)) in ("; 53019 + end; 53020 + if Strip(upcase(Selected)) eq 'YES' then do; 53021 + string = '"'!!trim(left(VARIABLE))!!'"'; 53022 + put string; 53023 + end; 53024 + if eof then do; 53025 + put ') then ROLE="INPUT";'; 53026 + put 'else ROLE="REJECTED";'; 53027 + put 'end;'; 53029 + %if %upcase(&EM_PROPERTY_HIDEVARIABLE) eq Y %then %do; 53030 + put 'if upcase(strip(ROLE)) = "REJECTED" then delete ;'; 53031 + %end; 53032 + end; 53033 + run; 53034 + quit; 53036 + filename X; 53037 + quit; 53038 +%mend MakeDeltaCode2; 53040 +%macro getInitialGClusterNumber(indata=, ninput=, ndummy=0, div=100, ngc=); 53041 + %global &ngc; 53042 + data _null_; 53043 + %if &indata ne %then %do; 53044 + %let dsid = %sysfunc(open(&indata)); 53045 + %let nvar = %sysfunc(attrn(&dsid, NVAR)); 53046 + %let dsid = %sysfunc(close(&dsid)); 53047 + %end; 53048 + %else %do; 53049 + %let nvar = %eval(&ninput+&ndummy); ; 53050 + %end; 53051 + %let numgc = %eval(&nvar/&div+2); 53052 + %let &ngc = &numgc; 53053 + run; 53054 + quit; 53055 +%mend getInitialGClusterNumber; 53058 +%macro MakeGobalConstellData(indata=, outlink=, outnode=); 53059 +data &outlink(drop = Selected); 53060 + set &indata; 53061 + LINKID = _N_; 53062 + label LINKID = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_linkid, noquote))" ; 53063 +run; 53064 +data &outnode(keep=NODEID TYPE LABEL); 53065 + set &indata; 53066 + length TYPE $16; 53067 + rename VARIABLE = NODEID; 53068 + *label CLUSNAME="Node ID"; 53069 + label CLUSNAME= "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_nodeidvar, noquote))" ; 53070 + TYPE = "VARIABLE"; 53071 + *label TYPE = "Node Type"; 53072 + label TYPE = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_nodetype, noquote))" ; 53073 + run; 53074 +data _tmp(keep=NODEID TYPE LABEL); 53075 + set &indata; 53076 + length TYPE $16; 53077 + rename CLUSNAME = NODEID; 53078 + label CLUSNAME= "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_nodeidvar, noquote))" ; 53079 + TYPE = "GCLUSTER"; 53080 + label TYPE = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_nodetype, noquote))" ; 53081 + run; 53082 +proc sort data=_tmp; 53083 + by NODEID; 53084 +run; 53085 +data _tmp; 53086 + set _tmp; by NODEID; 53087 + if first.NODEID then output; 53088 +run; 53089 +proc sort data=&outnode; 53090 + by NODEID; 53091 +run; 53092 +data &outnode; 53093 + set _tmp &outnode; 53094 +run; 53095 +proc datasets lib = work nolist; 53096 + delete _tmp; 53097 +run; 53098 +quit; 53099 +%mend MakeGobalConstellData; 53101 +/* Make contellation plot data among GCLUSTERS */ 53103 +%Macro MakeGClusterConstData(indata=, inoutrsq=, outnode=, outlink=); 53105 +data &outlink(keep = _NAME_ _PARENT_ _LABEL_ LINKID); 53106 + set &indata; 53107 + LINKID = _N_; 53108 + if upcase(substr(strip(_NAME_),1, 2))="CL" then do; 53109 + _NAME_ = "ROOT"||upcase(substr(strip(_NAME_),5)); 53110 + end; 53111 + if _PARENT_ ne " " and upcase(substr(strip(_PARENT_),1, 2))="CL" then do; 53112 + _PARENT_ = "ROOT"||upcase(substr(strip(_PARENT_),5)); 53113 + end; 53114 + if upcase(substr(strip(_LABEL_),1, 2))="CL" then do; 53115 + _LABEL_ = "ROOT"||upcase(substr(strip(_LABEL_),5)); 53116 + end; 53117 +run; 53119 +data _tmp_outrsquare; 53120 + set &inoutrsq; 53121 + if upcase(strip(TYPE)) = 'CLUSTERCOMP' then delete; 53122 +run; 53124 +proc freq data =_tmp_outrsquare noprint; 53125 + tables GCluster/out=_tmp_GCLUSFREQ(rename=(GCLUSTER=_NAME_)); 53126 +run; 53128 +data &outnode(keep=_NAME_ TYPE LABEL); 53129 + set &outlink; 53130 + length TYPE $16; 53131 + length LABEL $100; 53132 + /*label CLUSNAME= "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_nodeidvar, noquote))" ;*/ 53134 + if upcase(substr(strip(_NAME_),1, 2))='GC' then do; 53135 + TYPE = "GCLUSTER"; 53136 + LABEL = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_gcluster, noquote)):"||_NAME_; 53137 + end; 53138 + else do; 53139 + TYPE= "ROOT"; 53140 + LABEL= _NAME_; 53141 + end; 53142 + label TYPE = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_nodetype, noquote))" ; 53143 + label LABEL = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_label, noquote))" ; 53144 + label _NAME_ = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_nodeidvar, noquote))" ; 53145 + run; 53147 +proc sort data=&outnode; 53148 + by _NAME_; 53149 +proc sort data=_tmp_GCLUSFREQ; 53150 + by _NAME_; 53151 +run; 53153 +data &outnode; 53154 + merge &outnode _tmp_GCLUSFREQ; by _NAME_; 53155 + if COUNT=. then COUNT=1; 53156 +run; 53158 +proc datasets lib = work nolist; 53159 + delete _tmp_outrsquare _tmp_GCLUSFREQ; 53160 +run; 53162 +quit; 53163 +%Mend MakeGClusterConstData; 53166 +%macro CreateGClusterScoreCode(indata=, globalclusid=, fileref=); 53168 + data _gscoretmpds; 53169 + set &indata; 53170 + if (_TYPE_ in ('SCORE' 'MEAN' 'STD') and _NCL_ = 1 ) or (_TYPE_ in ('MEAN' 'STD')); 53171 + if _TYPE_ = 'MEAN' then _NAME_='MEAN'; 53172 + if _TYPE_ = 'STD' then _NAME_='STD'; 53173 + if _TYPE_ = 'SCORE' then _NAME_ = "GC"||strip(&globalclusid); 53174 + DROP _TYPE_ _NCL_; 53175 + run; 53177 + /* %let gscorefile = %bquote(&EM_NODEDIR)&EM_DSEP.gclusterscore.sas; 53178 + GCluster Component &globalclusid ------ */ 53180 + filename _file_ "&fileref"; 53182 + data _null_; 53183 + %if &globalclusid eq 1 %then %do; 53184 + FILE _file_; 53185 + %end; 53186 + %else %do; 53187 + FILE _file_ MOD; 53188 + %end; 53190 + %let dsid = %sysfunc(open(work._gscoretmpds)); 53191 + %let nvar = %sysfunc(attrn(&dsid, NVAR)); 53192 + %let vn_name =%sysfunc(varnum(&dsid, _NAME_)); 53194 + %let k = 1; 53195 + %do %while(^%sysfunc(fetch(&dsid))); 53196 + %let _name = %sysfunc(getvarc(&dsid, &vn_name)); 53197 + %if &k > 2 %then %do; 53198 + put "&_name = 0 ; "; 53199 + %end; 53200 + %let k = %eval(&k+1); 53201 + %end; 53203 + %let rc = %sysfunc(rewind(&dsid)); 53204 + %do i= 2 %to &nvar; 53205 + %let _varname = %sysfunc(varname(&dsid, &i)); 53206 + %do %while(^%sysfunc(fetch(&dsid))); 53207 + %let _name = %sysfunc(getvarc(&dsid, &vn_name)); 53208 + %if &_name = MEAN %then 53209 + %let _mean = %sysfunc(getvarn(&dsid, &i)); 53210 + %else %if &_name = STD %then 53211 + %let _std = %sysfunc(getvarn(&dsid, &i)); 53212 + %else %do; 53213 + %let coeff = %sysfunc(getvarn(&dsid, &i)); 53214 + %let abscoeff = %sysfunc(abs(&coeff)); 53215 + %if &abscoeff > 0 %then %do; 53216 + put "&_name = &_name+&coeff * (&_varname - &_mean)/&_std;"; 53217 + %end; 53218 + %end; 53219 + %end; 53220 + %let rc = %sysfunc(rewind(&dsid)); 53222 + %end; 53224 + %let dsid= %sysfunc(close(&dsid)); 53225 + run; 53228 + proc datasets lib=work nolist; 53229 + delete _gscoretmpds; 53230 + run; 53231 + quit; 53233 +%mend CreateGClusterScoreCode; 53236 +%macro MakeGClusterCorrelation(Indata=, ngcluster=, gscorecode=, outrsquare=); 53238 + %EM_REGISTER(KEY=GSCORE, TYPE=DATA); 53239 + %EM_GETNAME(KEY=GSCORE, TYPE=DATA); 53240 + %EM_REGISTER(KEY=GSCORESTAT, TYPE=DATA); 53241 + %EM_GETNAME(KEY=GSCORESTAT, TYPE=DATA); 53242 + %EM_REGISTER(KEY=GSCORETREE, TYPE=DATA); 53243 + %EM_GETNAME(KEY=GSCORETREE, TYPE=DATA); 53244 + %EM_REGISTER(KEY=GSCORECORR, TYPE=DATA); 53245 + %EM_GETNAME(KEY=GSCORECORR, TYPE=DATA); 53246 + %EM_REGISTER(KEY=GSCORECORRPLOT, TYPE=DATA); 53247 + %EM_GETNAME(KEY=GSCORECORRPLOT, TYPE=DATA); 53248 + %EM_REGISTER(KEY=GCLUSLINK, TYPE=DATA); 53249 + %EM_GETNAME(KEY=GCLUSLINK, TYPE=DATA); 53250 + %EM_REGISTER(KEY=GCLUSNODE, TYPE=DATA); 53251 + %EM_GETNAME(KEY=GCLUSNODE, TYPE=DATA); 53253 + filename gsfile "&gscorecode"; 53255 + data &EM_USER_GSCORE; 53256 + set &indata; 53257 + %include gsfile; 53258 + keep 53259 + %do i=1 %to &ngcluster; 53260 + %let gcvarname = GC&i; 53261 + &gcvarname 53262 + %end; 53263 + ; 53264 + run; 53266 + proc varclus data=&EM_USER_GSCORE outstat=&EM_USER_GSCORESTAT outtree=&EM_USER_GSCORETREE 53267 + %if %upcase(&EM_PROPERTY_CLUSCOMP) eq CENTROID %then %do; centroid %end; 53268 + %if %upcase(&EM_PROPERTY_CLUSSOURCE) eq COV %then %do; cov %end; 53269 + %if %upcase(&EM_PROPERTY_CLUSHIERACHY) eq Y %then %do; hi %end; 53270 + noprint ; 53271 + var 53272 + %do i=1 %to &ngcluster; 53273 + %let gcvarname = GC&i; 53274 + &gcvarname 53275 + %end; 53276 + ; 53277 + run; 53280 + %MakeVarClusCorrData(statds=&EM_USER_GSCORESTAT, corrds=&EM_USER_GSCORECORR, corrplotds=&EM_USER_GSCORECORRPLOT ); 53281 + data &EM_USER_GSCORECORRPLOT ; 53282 + set &EM_USER_GSCORECORRPLOT; 53283 + rename _X_ = X; 53284 + rename _Y_ = Y; 53285 + label _X_ = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_gcluster, noquote))" ; 53286 + label _Y_ = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_gcluster, noquote))" ; 53287 + run; 53289 + %MakeGClusterConstData(indata=&EM_USER_GSCORETREE, inoutrsq=&outrsquare, outnode=&EM_USER_GCLUSNODE, outlink=&EM_USER_GCLUSLINK); 53291 + data &EM_USER_GSCORETREE; 53292 + length _NAME_ $32; 53293 + length _LABEL_ $100; 53294 + set &EM_USER_GSCORETREE(DROP=_LABEL_); 53295 + if upcase(substr(strip(_NAME_),1, 2))='GC' then do; 53296 + _LABEL_ = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_gcluster, noquote)):"||_NAME_; 53297 + end;else do; 53298 + _LABEL_ = _NAME_; 53299 + end; 53300 + label _LABEL_ = "%sysfunc(sasmsg(sashelp.dmine, rpt_varclus_label_variable, noquote))"; 53302 + run; 53304 + quit; 53306 +%mend MakeGClusterCorrelation; NOTE: %INCLUDE (level 1) ending. NOTE: Fileref TEMP has been deassigned. 53309 53310 *------------------------------------------------------------*; 53311 * End CLOSEVARIABLESET: VarClus; 53312 *------------------------------------------------------------*; 53313 %em_delMacro; 203 The SAS System 10:11 Thursday, July 31, 201 4 53315 %let SYSCC=0; 53316 %let SYSRC=0; 53317 %let EMEXCEPTIONSTRING=; 53318 %let SYSMSG=; 53319 proc datasets lib=WORK nolist; delete COLUMNMETA;run; NOTE: The file WORK.COLUMNMETA (memtype=DATA) was not found, but appears on a DELETE statement. 53319 ! quit; NOTE: PROCEDURE DATASETS used (Total process time): real time 0.00 seconds cpu time 0.00 seconds 204 The SAS System 10:11 Thursday, July 31, 201 4 53320 %let SYSCC=0; 53321 %let SYSRC=0; 53322 %let EMEXCEPTIONSTRING=; 53323 %let SYSMSG=; 53324 %em_diagram(action=setproperties, projpath=%nrstr(/sasdata/EMiner_Projects), projname=%nrstr(31072014), dgmId=EMWS1, requestFile=DiagramSetPropertiesRequest.xml); NOTE: %INCLUDE (level 1) file _DGMFRF is file SASHELP.EMWIP.EM_DSEP.SOURCE. 53325 +%macro em_dsep; 53326 + %global emdsep; 53327 + %if %substr(&sysscp, 1, 3)= WIN %then 53328 + %let emdsep=\; 53329 + %else 53330 + %if %substr(&sysscp, 1, 3)= DNT %then 53331 + %let emdsep=\; 53332 + %else 53333 + %let emdsep=/; 53334 +%mend em_dsep; 53335 +%em_dsep; NOTE: %INCLUDE (level 1) ending. NOTE: Fileref _DGMFRF has been deassigned. WIP_ACTION: CLOSE DGMID: EMWS1 LOCKFILE: /sasdata/EMiner_Projects/31072014/Workspaces/EMWS1/System/wsopen.lck 53336 %global EM_REFRESH_PROPERTY; 53337 %let EM_REFRESH_PROPERTY=N; 53338 %let syscc=0; 53339 data WORK._EMVISUALPROPERTIES; 53340 length NODEID CLASS COMPONENT $32 X Y 8 LABEL $81; 53341 NODEID = "Trans2"; 53342 CLASS=''; 53343 component=''; 53344 X = 876; 53345 Y= 328; 53346 LABEL = "OPT"; 53347 output; 53348 NODEID = "Varsel"; 53349 CLASS=''; 53350 component=''; 53351 X = 1202; 53352 Y= 229; 53353 LABEL = "Variable Selection"; 53354 output; 53355 NODEID = "DMDB2"; 53356 CLASS=''; 53357 component=''; 53358 X = 1076; 53359 Y= 329; 53360 LABEL = "DMDB"; 53361 output; 53362 NODEID = "DMDB"; 53363 CLASS=''; 53364 component=''; 53365 X = 1046; 53366 Y= 177; 53367 LABEL = "DMDB"; 53368 output; 53369 NODEID = "Impt"; 53370 CLASS=''; 53371 component=''; 53372 X = 727; 53373 Y= 9; 53374 LABEL = "Impute"; 53375 output; 53376 NODEID = "Varsel2"; 53377 CLASS=''; 53378 component=''; 53379 X = 1229; 53380 Y= 399; 53381 LABEL = "Variable Selection"; 53382 output; 53383 NODEID = "Stat"; 53384 CLASS=''; 53385 component=''; 53386 X = 546; 53387 Y= 9; 53388 LABEL = "StatExplore"; 53389 output; 53390 NODEID = "Trans"; 53391 CLASS=''; 53392 component=''; 53393 X = 875; 53394 Y= 177; 53395 LABEL = "BEST"; 53396 output; 53397 NODEID = "Ids2"; 53398 CLASS=''; 53399 component=''; 53400 X = 205; 53401 Y= 523; 53402 LABEL = "OZ_MODEL_2107"; 53403 output; 53404 NODEID = "Ids"; 53405 CLASS=''; 53406 component=''; 53407 X = 54; 53408 Y= 9; 53409 LABEL = "OZ_MODEL_2107"; 53410 output; 53411 NODEID = "Part"; 53412 CLASS=''; 53413 component=''; 53414 X = 375; 53415 Y= 9; 53416 LABEL = "Data Partition"; 53417 output; 53418 NODEID = "Smpl"; 53419 CLASS=''; 53420 component=''; 53421 X = 211; 53422 Y= 9; 53423 LABEL = "Sample"; 53424 output; 53425 NODEID = "VarClus2"; 53426 CLASS=''; 53427 component=''; 53428 X = 1241; 53429 Y= 329; 53430 LABEL = "Variable Clustering"; 53431 output; 53432 NODEID = "VarClus"; 53433 CLASS=''; 53434 component=''; 53435 X = 1199; 53436 Y= 178; 53437 LABEL = "Variable Clustering"; 53438 output; 53439 run; NOTE: The data set WORK._EMVISUALPROPERTIES has 14 observations and 6 variables. NOTE: DATA statement used (Total process time): real time 0.00 seconds cpu time 0.02 seconds 53440 proc sort data=WORK._EMVISUALPROPERTIES; 53441 by NODEID; 53442 run; NOTE: There were 14 observations read from the data set WORK._EMVISUALPROPERTIES. NOTE: The data set WORK._EMVISUALPROPERTIES has 14 observations and 6 variables. NOTE: PROCEDURE SORT used (Total process time): real time 0.00 seconds cpu time 0.00 seconds 53443 proc sort data=EMWS1.em_nodeid out=_tempNodeid; 53444 by NODEID; 53445 run; NOTE: There were 14 observations read from the data set EMWS1.EM_NODEID. NOTE: The data set WORK._TEMPNODEID has 14 observations and 8 variables. NOTE: PROCEDURE SORT used (Total process time): real time 0.00 seconds cpu time 0.00 seconds 53446 data _tempNodeid; 53447 update _tempNodeid(in=_a) WORK._EMVISUALPROPERTIES(in=_b); 53448 by NODEID; 53449 if _a then output; 53450 run; NOTE: There were 14 observations read from the data set WORK._TEMPNODEID.

SIlver_Geo · ‎05-13-2014

Dear Miguel, Thank you for your assistance, I made quite an extensive reading and I was able to understand the whole process. I used in the end only WoE variables since in behavioural credit scoring can make more sense and show the risk for every variable. Best Regards, George

kkasahar · ‎05-08-2014

Hi Miguel, I really appreciate your advice. Below is the result of your suggestions: A) Gradient Boosting. (Data -> Gradient Boosting) Result: Model Misclassification rate ASE ROC Gini Coef. Y Boost Gradient Boosting 0.005701214 0.00566871 0.5 0 Model Model Description Data Role Target FALSE Negative TRUE Negative FALSE Positive TRUE Positive Node Boost Gradient Boosting TRAIN RESPONSE_IND 6445 1124016 0 0 This model classified all cases as non-response – just like other models that I built – because my response rate is (0.0057 or 6445 cases) and non-response is (0.9943 or 1124016) and True Positive and False Positive are both zero. B) Ensemble of Reg2 and Reg3 (using 10% oversampled data) Selected Model Model Node Valid: Average Profit for RESPONSE_IND Train: Average Squared Error Train: Misclassification Rate Valid: Average Squared Error Valid: Misclassification Rate Y Reg2 1.71 0.090012 0.10002 0.089988 0.099984 Reg3 1.71 0.090012 0.10002 0.089988 0.099984 Ensmbl 1.71 0.098908 0.10002 0.098877 0.099984 C) Bagging, Boosting, and Rotation Forest. This paper you suggested me is great!! I am still working on this part since I have not used these methods. I will let know. Findings from my search ( I am not sure how reliable they are) I was reading SAS support page and they said, “Over-weighting or under-sampling can improve predictive accuracy when there are three or more classes, including at least one rare class and two or more common classes.” My data has two classes, so I guess it is true for my case because my oversampling (omitting cases from common classes) is not helping to obtain the best model regardless of sample sizes. However, when I don't put adjusted prior (using the prior of sample data), my misclassifications increases as I increase sample size as well as the number of True Positive. I care more about the number of True Positive rather than errors. Is it wrong if I keep current prior probablity(not using adjusted prior)? Below is the link to the website. http://support.sas.com/documentation/cdl/en/emxndg/64759/HTML/default/viewer.htm#p1w6fewo0jhzxdn1rytuk1kt0pqj.htm The other thing is how to evaluate models when you need to detect the rare event. Detection rate (Recall) - ratio between the number of correctly detected rare events and the total number of rare events False alarm (false positive) rate – ratio between the number of data records from majority class that are misclassified as rare events and the total number of data records from majority class. ROC Curve is a trade-off between detection rate and false alarm rate so Missclassification, ASE, Average Profit , etc are not sufficient metric for evaluation when rare event?? Here is the title of article: "Data Mining for Analysis of Rare Events: A Case Study in Security, Financial and Medical Applications" Thanks, Miguel! I really appreciate your help.

SIlver_Geo · ‎05-07-2014

I found AUC it is called ROC index. My only issue now is regarding test set or validation set usage. From what I read is not the same. There should be a way to see the results for the test set.

MarianaL · ‎04-02-2014

OK, I finally figured that out. In addition to all of the above settings, I put the Minimum Categorical Size =1 and now I have all values.

Lychee · ‎04-01-2014

This is my first time to come to this site. I can't believe that I learned so much. Thank you very much, Jaap.

KimmoSivonen · ‎03-31-2014

Thank you Miguel! Best Wishes! Kimmo

Online Status	Offline
Date Last Visited	‎02-28-2018 11:39 AM

Re: Unbalanced data - miner

Re: SAS EM only: How to use parameter estimates in the next node?

Re: StatExplore Node

Re: How many leaves and nodes should a tree

Re: Export scoring code for Cross Validation in SAS Enterprise Miner

Re: Export scoring code for Cross Validation in SAS Enterprise Miner

Re: run time error ensemble model

Re: run time error ensemble model

Re: help with hash table

Re: help with hash table

Re: StatExplore Node

Re: How to access Variable importance in neural network in EM?

Re: Grouping variables to create new variables SAS Enterprise Miner

Re: Error when running market basket node in SAS EM

Re: Seed Initialization Method for Hierarchical Clustering

Re: Using cross-validation in Enterprise Miner;

Re: How come no Segment Profile after I set "Cluster Variable Role" = ...

Re: Confusion matrix in Enterprise Miner

Re: How can we export dataset from enterprise Miner as a csv file or t...

Re: How many leaves and nodes should a tree

Credit Scoring by Example in SAS® Enterprise Miner™

Tip: How to model a rare target using an oversample approach in SAS® ...

Tip: How to interpret your SAS® Rapid Predictive Modeler results

Tip: Use the Cutoff Node in SAS® Enterprise Miner™ to Consume the Post...

Tip: How to build a scorecard using Credit Scoring for SAS® Enterprise...

Re: How to create year series?

Re: Oversampling in Enterprise Miner with a rare event fixed proportio...

Re: Cross Validation type in Enterprise Miner

Re: Final Bagged Decision Tree from group processing?

Re: Confusion matrix in Enterprise Miner

Re: Capping and Flooring outliers - Methods

Re: How can we export dataset from enterprise Miner as a csv file or t...

Re: Classification: K nearest neighbors (MBR)

Re: Decision Tree Node creating no D_TARGET outcome

Re: Usage of WoE or Grouping Variables In Regression Node

Re: True Positive is 0. Oversampling, adjusted prior, weight. Need a ...

Re: Classification Matrix on Test Set

Re: SAS EM Node Rules code is omitting data records

Re: How to explore a dataset with over 3 million rows?

Re: SAS EM Reporter Node and Scorecard Points