M_Maldonado Tracker

Re: Unbalanced data - miner

M_Maldonado — Wed, 28 Feb 2018 15:51:00 GMT

Hi Nicolas,

Maybe this thread can help you while someone takes a second look into what you did?

https://communities.sas.com/t5/SAS-Data-Mining-and-Machine/Oversampling-in-Enterprise-Miner-with-a-rare-event-fixed/td-p/161991

When I oversample, I usually test the model on a hold-out test data set that I saved somewhere else and didn't use for modeling. That gives me some confidence that I didn't fool myself 🙂
Would that be an option for you?

Best,
-Miguel

Re: SAS EM only: How to use parameter estimates in the next node?

M_Maldonado — Fri, 17 Nov 2017 17:17:39 GMT

Hey Mike,

Not sure if I am reading your question right. But I think that you are asking about training a model with a few observations and using that model to score a larger data set.

There are a couple ways to do this and it all depends on whether you have one big data set that you want to split into Train/Validate/Testing or if you are going to import 2 data sets (one to be splitted into Training/Validation and the other one get scored by a score node).

From your comment about "Simply hooking the nodes up doesn't work - where the full regression node has two input" it sounds like you are trying to do this with 2 data sets. To do this you will use your smaller data set to train/validate your model, and after that you will use a Score node to score your larger data set. You will see quick examples on how to do this on the Score node section.

Give it a try, and if you get into any trouble, add a screenshot of your diagram to make it easier to help you out.

Good luck!

Re: StatExplore Node

M_Maldonado — Mon, 05 Dec 2016 05:44:00 GMT

Oi Joao, tudo bem?

You will see a few Stats for your variables when you scroll down the Output window. Not sure you get all the ones on your list, but you should get some of those for your interval variables.

Does that help?

Boa sorte!

-Miguel

Re: How many leaves and nodes should a tree

M_Maldonado — Mon, 05 Dec 2016 05:34:29 GMT

Hey--if you haven't, check out the Getting Started with EM series.

In this one Chip Robie talks about decision trees and you will see a brief screensots of the Interactive Decision trees, starting at 9:15.

Good luck!

-Miguel

https://youtu.be/IlUZYlgkeSc?t=9m15s

Re: Export scoring code for Cross Validation in SAS Enterprise Miner

M_Maldonado — Sun, 16 Oct 2016 23:38:59 GMT

I like that workaround!

XML of the diagram or a quick screenshot, or both 🙂

When you have a chance, I am also very curious to know more about your learnings about cross validation. In particular, do you feel like you get more predictive power, or anything else you might share?

Thanks!

Re: Export scoring code for Cross Validation in SAS Enterprise Miner

M_Maldonado — Fri, 14 Oct 2016 16:51:10 GMT

Hi David,

Sorry I am late to the party.

I don't have EM handy. Sadly I spend more time in meetings than on hands-on software these days.

This is the kind of thing that I would suggest fixing directly on the score code while someone figures out the right way to do this.

If you have a chance, post the score code of the flow you have (the simpler the data the better), and the community and myself will give you suggestions!

Best,

-M

Re: run time error ensemble model

M_Maldonado — Wed, 20 Apr 2016 13:36:38 GMT

It looks like for some reason the metadata from your flow is not getting correctly to the Start Groups node.

I'd try to shortcut the File Import. For example do File Import->Save Data node.

Then import the data you saved with the wizard to add it as a data source. Bit of a pain, I know, but that should fix it.

Good luck!

Re: run time error ensemble model

M_Maldonado — Tue, 19 Apr 2016 12:51:48 GMT

Hello,

Not sure I am reading this right. Can you share a screenshot of your diagram?

Thanks,

Re: help with hash table

M_Maldonado — Wed, 13 Apr 2016 14:54:37 GMT

Hi Xia,

Nice program! I didn't get it to work but I wasn't sure is if I my SAS version is too old (9.1.3) or if &colon is a macro variable that I had to declare somewhere with a %let statement.

I will give it another try with SAS 9.4 which I am so grateful to get shortly. I'll start on 9.4 to leverage hpsplit, gampl, and logistic procedures for our predictive modeling. With the extra revenue we'll buy a full license of Enterprise Miner next year.

I call myself a SAS alumni. Part of SAS family once, means SAS family always.

Thanks again! Looking forward to trying your code on SAS 9.4.

Keeping a copy of the log here for my records...


1
2    data have(rename=(old=from contract=to));
3    contract='1';old='';output;
4    contract='2';old='1';output;
5    contract='x';old='';output;
6    contract='a';old='';output;
7    contract='3';old='2';output;
8    contract='y';old='';output;
9    contract='b';old='a';output;
10   contract='4';old='3';output;
11   contract='z';old='';output;
12   contract='5';old='4';output;
13   contract='c';old='b';output;
14   contract='d';old='c';output;
15   run;

NOTE: The data set WORK.HAVE has 12 observations and 2 variables.
NOTE: DATA statement used (Total process time):
      real time           0.01 seconds
      cpu time            0.01 seconds


16
17
18
19   data full;
20     set have end=last;
21     if _n_ eq 1 then do;
22      declare hash h();
23       h.definekey('node');
24        h.definedata('node');
25        h.definedone();
26     end;
27     output;
28     node=from; h.replace();
29     from=to; to=node;
30     output;
31     node=from; h.replace();
32     if last then h.output(dataset:'node(where=(node is not missing))');
33     drop node;
34   run;

ERROR: Invalid data set name at line 32 column 16.
ERROR: An error has occurred during instance method OM_OUTPUT(505) of "DATASTEP.HASH".
NOTE: The SAS System stopped processing this step because of errors.
NOTE: There were 12 observations read from the data set WORK.HAVE.
WARNING: The data set WORK.FULL may be incomplete.  When this step was stopped there were 24
         observations and 2 variables.
NOTE: DATA statement used (Total process time):
      real time           0.01 seconds
      cpu time            0.01 seconds


35
36
37   data want(keep=node household);
38   declare hash ha(ordered:'a');
39   declare hiter hi('ha');
40   ha.definekey('count');
41   ha.definedata('last');
42   ha.definedone();
43   declare hash _ha(hashexp: 20);
44   _ha.definekey('key');
45   _ha.definedone();
46
47   if 0 then set full;
48   declare hash from_to(dataset:'full(where=(from is not missing and to is not
48 ! missing))',hashexp:20,multidata&colon:'y');
                                          -
                                          22
                                          76
WARNING: Apparent symbolic reference COLON not resolved.
ERROR 22-322: Syntax error, expecting one of the following: <, <=, =, >, >=, EQ, GE, GT, LE, LT,
              NE, NG, NL, ^=, ~=.

ERROR 76-322: Syntax error, statement will be ignored.

49    from_to.definekey('from');
50    from_to.definedata('to');
51    from_to.definedone();
52
53   if 0 then set node;
ERROR: File WORK.NODE.DATA does not exist.
54   declare hash no(dataset:'node');
55   declare hiter hi_no('no');
56    no.definekey('node');
57    no.definedata('node');
58    no.definedone();
59
60
61   do while(hi_no.next()=0);
62    household+1; output;
63    count=1;
64    key=node;_ha.add();
65    last=node;ha.add();
66    rc=hi.first();
67    do while(rc=0);
68      from=last;rx=from_to.find();
69      do while(rx=0);
70        key=to;ry=_ha.check();
71         if ry ne 0 then do;
72          node=to;output;rr=no.remove(key:node);
NOTE: DATA statement used (Total process time):
      real time           0.00 seconds
      cpu time            0.00 seconds

ERROR: DATA STEP Component Object failure.  Aborted during the COMPILATION phase.
NOTE: The SAS System stopped processing this step because of errors.
73          key=to;_ha.add();
74          count+1;
75          last=to;ha.add();
76         end;
77         rx=from_to.find_next();
              -----------------
              558
ERROR 558-185: Reference find_next is not a member of object from_to.

78      end;
79      rc=hi.next();
80   end;
81   ha.clear();_ha.clear();
82   end;
83   stop;
84   run;

Re: help with hash table

M_Maldonado — Tue, 12 Apr 2016 18:42:57 GMT

Uncommenting this line did the trick:

*if sc.find() ge 0;

It also works if you assign that find() value to any variable. E.g.

code=sc.find();

help with hash table

M_Maldonado — Tue, 12 Apr 2016 15:32:43 GMT

Hi,

I want to use a hash table to create a lookup table.

Can someone help me correct the below?

Long story short, I am going to loop over the new and old contracts of a given customer. I got this working with data step, but it is too slow. Will give it a try with hash tables.

For the example below, once I iterate over these cases, I will notice that I have 5 customers:

C1 had contracts 1,2,3,4,5

C2 had contracts a,b,c,d

C3,C4,C5 respectively had contracts x,y,z

Thanks!

Miguel

data a;
contract='1';old='';output;
contract='2';old='1';output;
contract='x';old='';output;
contract='a';old='';output;
contract='3';old='2';output;
contract='y';old='';output;
contract='b';old='a';output;
contract='4';old='3';output;
contract='z';old='';output;
contract='5';old='4';output;
contract='c';old='b';output;
contract='d';old='c';output;
run;

data b(rename=(old=contract contract=contract1));
set a;
run;

data result;
length contract contract1 $ 1;
set a;
if _n_=1 then do;
 declare hash sc (dataset: 'b');
 sc.definekey('contract');
 sc.definedata('contract1');
 sc.defineDone();
end;
*if sc.find() ge 0;
run;

/*The results data set is not what I expected*/
/*work.expected below is what I am trying to get*/

proc sort data=a;
by contract;
run;
proc sort data=b;
by contract;
run;

data expected;
merge a b;
by contract;
run;

Re: Why ensemble is decreasing my overall lift?

M_Maldonado — Fri, 04 Mar 2016 14:49:21 GMT

My bad, I don't know how to count... I meant to ask, the Misc of your 4 models and their ensemble.

I am curious if the ensemble of all 4 models is getting worse--and if the Tree or the SVM are tripping it off...

Since you are at it, can you include the classification charts as well?

Maybe that will give us a better light of what's going on.

Thanks!

Re: Churn Prediction variables for post paid data

M_Maldonado — Fri, 04 Mar 2016 14:46:21 GMT

Hi Sarbarup,

From a competition I did not do too well at, these were the predictors:

account length
international plan
voice mail plan
number of voice mail messages
total day minutes used
day calls made
total day charge
total evening minutes
total evening calls
total evening charge
total night minutes
total night calls
total night charge
total international minutes used
total international calls made
total international charge
number customer service calls made

More info:

CrowdAnalytix - why customers churn?

https://www.crowdanalytix.com/contests/why-customer-churn/

Good luck!

-Miguel

Re: Why ensemble is decreasing my overall lift?

M_Maldonado — Thu, 03 Mar 2016 17:28:34 GMT

Hi Munitech,

Right, maximum will take the maximum posterior probability of your models.

What is the misclassification for your 3 models and their ensemble?

Thanks,

Re: Help finding enterprise datasets for process mining research

M_Maldonado — Sun, 07 Feb 2016 22:41:58 GMT

Hi Yeti,

Not sure if I get the right idea of your example scenario. What would this data look like (inputs, target, ids, etc)?

Show us a concrete mockup of this data. Maybe there is something similar from a SAS course or a data set in sashelp or sampsio libraries.

In the meantime, this book is a go-to for research and proofs of concept:

Simulating data with SAS

by Rick Wicklin

Good luck!

-Miguel

Re: Proc Treeboost

M_Maldonado — Sun, 07 Feb 2016 22:32:40 GMT

Hi Todd,

Are you talking about the score code or optimized score code produced by Gradient Boosting node?

Where do you see this gamma parameter?

Re: Interactive Grouping results are empty

M_Maldonado — Tue, 02 Feb 2016 16:00:17 GMT

Hi iae,

Is there anything special about this data set? Is the target a rare event?

Try different settings for binning. Some suggestions below.

-bump up to 50 the number of bins for interval variable binning.

-grouping method (default vs constrained optimal)

-change tree based grouping options for all methods but constrained optimal

-check the log when you run with constrained optimal to confirm what constraints are not holding, and change them (constrained optimal or advanced constrined optimal options).

-try Apply restrictions No to see if this helps and then decide whether you need this or not.

More info please

A couple screenshots of Interactive Grouping, specially what does the grouping plots on interactive mode (event count vs pre-bins and event counts vs groupings) look like? Any WOE trend whatsoever?

thanks!

Re: Stock price indicators

M_Maldonado — Mon, 01 Feb 2016 16:28:39 GMT

Hi Illidan,

What have you put together so far?

I gave @Lenon one idea about stocks on this thread: https://communities.sas.com/t5/SAS-Data-Mining/stock-price-Run-up/m-p/204145/highlight/true#M2750

Maybe he can share more about his methods?

Re: misclassification rate sas miner

M_Maldonado — Mon, 01 Feb 2016 12:10:53 GMT

Hi Rogelio,

It looks like most fit statistics are defined only once on the Reference Help (press F1 when you are on Enterprise Miner to see the Reference Help). If they are defined on the Model Comparison node, you don't see them again on the Scorecard node, unless they are a specific result of the Scorecard node.

I hope this helps!

Re: SAS Miner: Filter Node: Can't exclude element of small size by selecting bar chart.

M_Maldonado — Thu, 28 Jan 2016 14:59:38 GMT

One way to do it. Add a Transform Variables node. Click on the SAS Code ellipsis and then write something like the suggestions below.

if myvariable="X" then delete;

if myvariable="X" then myvariable="";

etc.

I hope this helps!