jaredp Tracker

Re: EM Decision Trees - Stratification or Not - Validation or Test?

jaredp — Wed, 19 Nov 2014 17:06:03 GMT

Hi Zachary. In the words of Spiderman, my spider sense is tingling.

I hope there are no differences in data set A (n=50,000) and data set B (n=25,000). I've been in this situation before and was told data set B was collected the same as data set A, but I find out later there was a slight methodological change in collected for B. Yes, the variables in both data sets are the same, but the underlying values had different assumptions.

You wouldn't want to validate a model using nuanced data.

Maybe there is a way to do both approaches and compare them. If you had a third data set for scoring, you could compare the results of the models.

Re: Tip: Use the Control Point Node for Simpler, Reusable Process Flows

jaredp — Wed, 05 Nov 2014 16:19:20 GMT

Awesome. After using EM for quite a few years, I only just discovered this node a little over a year ago. It is a big help! But it never occurred to me to put it at the start of my flows like you did! Thanks for sharing!

Re: How to convert 1 as one and 2 as two without Proc format

jaredp — Thu, 15 May 2014 14:46:23 GMT

A "correct answer" has been provided. Did it not work for you?

Re: Suggest other ways of creating a frequency variable

jaredp — Mon, 28 Apr 2014 22:24:10 GMT

This one runs a little slower, but it's an attractive option since everything is in one package. It's very tempting to use because of that. For my immediate project, I'll stick with one fellows suggestion, but your proc sql code is going to help me in some adhoc stuff. I like it!

I like that you spotted my comment in the code about not requiring the data be ordered

Thank you!

Re: Suggest other ways of creating a frequency variable

jaredp — Mon, 28 Apr 2014 22:19:04 GMT

Although I selected the other response as the correct answer, using proc freq actually seems to be slightly faster than proc summary. Both are faster than proc tabulate.

So I think I'll be using your proc freq suggestion. It's funny I didn't use proc freq in the first place, since it's the one I normally use. But back when I did this code, I was, admittedly, being lazy (or pressured for results...I can't remember).

Thanks!

Re: Suggest other ways of creating a frequency variable

jaredp — Mon, 28 Apr 2014 22:15:32 GMT

Wow, I had a longer response, but I timed out because I was busy testing everyone's code suggestions with my actual data.

That's a good point about not using Tabulate. In truth, I used it because at the time I only had to copy, paste and tweak some of my existing code from another SAS program. I admit I was being lazy, but isn't that one of the traits of being a programmer?

I'm looking now at putting this into a more production environment, so that's why optimizing is important. For me (or at least in the environment I work in), part of optimization takes into account code maintenance. Not only is your solution speedy with my data set, it's also easy on the eyes for good code maintenance.

Thank you!

Suggest other ways of creating a frequency variable

jaredp — Mon, 28 Apr 2014 16:54:00 GMT

Hi everyone.

I'm hoping to benefit from your expertise to learn of new approaches to my solution below. The solution below works for me, although the proc tabulate feels a little slow at times on my larger data sets. Perhaps doing the entire solution in proc sql could be faster? But what would that look like? Or maybe a hash solution could be used here? Any thoughts or code is appreciated.

data have;

input name $;

cards;

john

jane

jack

jill

jenn

jeff

john

jill

jenn

jack

jane

jack

jane

jenn

;

run;

/* the frequency of the name is put into the totalcount variable */

data want; /*sorted or unsorted doesn't really matter*/

input name $ totalcount;

cards;

john 3

jane 3

jack 5

jill 2

jenn 4

jeff 1

john 3

jill 2

jenn 4

jack 5

jane 3

jack 5

jane 3

jenn 4

;

run;

proc tabulate data=have out=haveN order=freq;/*order=data will sort by name*/

class name;

tables name, N;

run;

proc sql;

/* 'Inner Join';*/

create table want as

select h.*, hn.N

from have as h, haveN as hn

where h.name=hn.name;

quit;

Re: Churn Analysis with SAS Miner- Help

jaredp — Wed, 23 Apr 2014 17:53:05 GMT

Good suggestions so far. Laura's suggestion is very good.

With a binary variable, many analysts (and business users) like the output from the Decision Tree node. It can help you identify which variables might be important factors for churn and give you a model to score new data.

Here is some good reading:http://support.sas.com/publishing/pubcat/chaps/57587.pdf

A REALLY good book to pick up is http://www.amazon.com/Data-Mining-Techniques-Relationship-Management/dp/0470650931. It has a great chapter on decision trees and likely covers survival analysis too.

Your data set has character variables that I *think* should be numeric. For example, Revenue would look like 22.000.000 which I think should have been $22,000,000 (or 22000000)?

When you import the data into EM, make sure you spend the time to set the roles and levels of each variable. Churn would be Target and Binary. There are a number of other variables that should be set to Binary too.

Customer could be set to a role of ID.

ChurnDep should be Rejected because it is redundant to churn. The decision tree will split once on that variable and stop splitting.

Calibrat should also likely be rejected.

Re: SAS X Command won't produce log file on linux server

jaredp — Wed, 23 Apr 2014 15:08:33 GMT

A couple of years back I was having issues running X commands from EG in a desktop environment. Some links on the topic you might be interested in:

http://blogs.sas.com/content/sasdummy/2009/11/19/using-the-x-and-systask-commands-from-sas-enterprise-guide/

41058 - Unable to submit X commands from SAS® Enterprise Guide® connecting to a SAS® Metadata Server when running SAS® 9.2 or later

https://groups.google.com/forum/#!msg/comp.soft-sys.sas/ezaLXdRo_c0/Y9mBLA4ScFMJ

NOXCMD: NO eXternal CoMmanDs! - platformadmin.com

Re: What's the difference between the rule builder and content categorization

jaredp — Thu, 17 Apr 2014 20:30:08 GMT

The book "Text Mining and Analysis: Practical Methods, Examples, and Case Studies ... By Goutam Chakraborty, Murali Pagolu, Satish Garla" explains your question quite well.

http://books.google.ca/books?id=SUKDAgAAQBAJ&pg=PT218&lpg=PT218&dq=text+rule+builder+node&source=bl&ots=GRW1z6ay_V&sig=UcTKkT5ecN1KcZs9QOEWTQ6HLqM&hl=en&sa=X&ei=kDdQU5GLDdKG2wWAsIH4DA&ved=0CFIQ6AEwCA#v=onepage&q=text%20rule%20builder%20node&f=false

Re: We REALLY ought to take more aggressive action against spammers!!!

jaredp — Fri, 04 Apr 2014 19:33:43 GMT

I can respect that. And you raise a good point about SGF and volunteers. Yet I think SGF and an online community are totally different from a volunteer perspective. But like I alluded to, I might be overly cautious due to past experiences. I'd be more than happy to be proven wrong on this one!

Re: We REALLY ought to take more aggressive action against spammers!!!

jaredp — Fri, 04 Apr 2014 16:09:19 GMT

"Reporting Abuse" is the best option, IMHO. This is a SAS run community and should be controlled by SAS alone. There is no telling what happens when you mix objectives (that is to say, your objectives of the site might not align with SAS' objectives).

SAS employees should have final say on matters of deleting or modifying content. As a community manager myself, I've seen what relinquishing control can do. Lines get crossed, mistakes are made, and power can go to people's heads.

I'll jokingly say that SAS should employ Text and Fraud Analytics to better identify spam and flag it before it goes live, but the more realistic options would be to keep the Report Abuse link and/or look into something like Akismet.

Re: How to convert 1 as one and 2 as two without Proc format

jaredp — Wed, 19 Feb 2014 18:12:07 GMT

This sounds like a homework assignment. :smileysilly:

Re: Import Data Error

jaredp — Wed, 19 Feb 2014 16:11:31 GMT

When I had this error, it had to do with my license. My license for SAS had expired. I was still able to load EG, but nothing in it would work.

Re: Standalone install of Eminer client 12.1 on Windows 8

jaredp — Wed, 12 Feb 2014 19:15:48 GMT

You might be installing the wrong thing. I *think* there are 2 versions of eminer. One is for the server environment, the other is for a desktop environment. I recall having to select the one I wanted. Choose the one that indicates a Workstation, if there is an option.

Re: Database dropped leading zeros...

jaredp — Mon, 25 Nov 2013 16:38:59 GMT

data _null_;

package_no=1234;

package_char = put(package_no, z7.);

put package_char;

run;

Re: Missing value for char variable-how to delete

jaredp — Mon, 04 Nov 2013 16:38:16 GMT

Could it also be that SAS is trying to convert the non-blank values in SIC?

Re: Weekend Challenge

jaredp — Mon, 04 Nov 2013 16:29:25 GMT

Not having SAS at home means working on this challenge would mean I'd have to work on the weekend.

Re: Text Categorization Rule

jaredp — Wed, 30 Oct 2013 16:38:50 GMT

I'm barely familiar with what you are doing above. I haven't got into the rule building stuff yet. But I did notice, in regards to your first question, that perhaps "call" is different from "Call" in the Rule column? Another guess I have is that one is a verb and the other is not?

I can't answer your second question, but I am interested in the answer. My gut feeling is that order shouldn't matter. If it did, then I'd think that the rules are not fine tuned enough? But perhaps the order of rules is a rule in and of itself....

Good questions...thanks for asking them.

Re: SAS Import CSV file - Free text column - carriage return

jaredp — Wed, 30 Oct 2013 16:12:29 GMT

Just thought I'd add some thoughts to what a few posters have already touched on.

One issue I see is that your example data is not a properly formatted CSV file. When I create CSV files, I encapsulate my free text fields with double quotations "like this". This tells most software that everything between the quotes is part of one column, regardless of what it contains. The only issue is when your free text field itself contains double quotes. This will cause software to cut off your data prematurely.

One option is to change how the source data is generated. Instead of using a comma, use a set of characters as your delimiter which are less likely to appear in your free text field, such as #*#*# . You then split your data based on that delimeter as opposed to splitting on a comma.

You could also try an approach similar to pradeepalankar . I personally like this approach. Get your first two and last two fields. Everything else is your third field.

It sounds like you have flexibility for what your final data set will look like. With the code already suggested, you easily aim for this data set:

Field1    Field2    Field3    Field4    Field5
001       12A       cards     UK        001,12A, Tues to Friday John rotates with Team, Perm 10800-1645 Sat Perm 8am starts Tue to Sat as of 22/02, cards, UK
002       12B       HL         UK        002,12B, Mon to Wed Marry rotates with Team, Perm 0800-1645 Sat Perm 8am starts Tue to Sat as of 22/02 Works in shift, HL, UK
003       12c        HL        UK         003,12c, Sat&Sun Paul rotates with Team, Perm 19000-1645 Sat,HL, UK
004       12D       CC        UK         004,12D, All day Joe rotates with PL Team, Perm 10800-1645 Sat 8am starts Tue to Sat as of 24/02 Works in shift, CC, UK