About geniusgenie

geniusgenie · ‎06-04-2017

Hi, I was wondering if someone can tell me how to perform 10-fold cross validation in 1. Decision Trees, 2. Random Forests 3. SVM 4. Neural Network 5. Logistic Regression etc... using SAS Enterprise Miner. Regards

geniusgenie · ‎06-04-2017

Hi Tom, Reason why I uploaded XLS file because in my orginal post I copied few rows of data which was not clear for Ballard. If you notice in sheet number two and three some columns contain long numeric values , these values are actually hexa decimal values. for these values I am looking to get a sum of unique values per file number. For columns like sd_type, td_sname, td_stype , I am looking to transpose them into columns. These columns will contain either 0 or 1 to represent whether this column was present in this file. I called it a flag (I may be wrong on this). I do not want concatenation. As I explained earlier, my main dataset contains only one row per file and these second and third datasets contain more than one rows. I need to merge them into main datasets in such a way that I just copy columns on the right hand side of the last column of main dataset. Because they contain unique file numbers.

geniusgenie · ‎06-03-2017

Hi Tom, I am looking to transpose rows into column but the columns containing hex values may be treated as sum of unqiue values per filename. And columns like td_sname or type can be treated as a flag in which 1 represents particular value present . So transpose will let mutiple columns to be present which is my case. Each file in ds2 and 3 contains atleast 3 or more rows so multiple features in a single row after transpose. Sample is in sheet4. Only thing i dont want is to skip any row unless same value gets repeated.

geniusgenie · ‎06-03-2017

Hi Tom, I am looking to transpose rows into column but the columns containing hex values may be treated as sum of unqiue values per filename. And columns like td_sname or type can be treated as a flag in which 1 represents particular value present . So transpose will let mutiple columns to be present which is my case. Each file in ds2 and 3 contains atleast 3 or more rows so multiple features in a single row after transpose. Sample is in sheet 4.

geniusgenie · ‎06-03-2017

Hi Ballard, Thanks for your reply, I am attaching sample XLS sheet for your reference of the datasets. Appologies for not creating dataset they way you asked, I am new to SAS so need study a little bit more. Sheet 1 contains main data in which I need to integrate sheet 2 and sheet 3. Its like 1 to many relationship. As sheet2 and 3 contain multiple rows of the same files present in sheet1. Sheet 2 and 3 contains fields highlighted as red, about them I am not sure how to integrate. One way I thought was to integrate like a count of unique addresses per file for example for file 1 , in sd_off the count will be 2 , sd_virtad count=3 etc. In sheet2, sd_type becomes the feature and represented as a yes/no value in integrated sheet. In sheet3, td_sname and td_stype are two important columns and have unique combinations of values as for every td_sname there is a td_stype, so, every unique combination becomes a column and represented as a yes/no value for each file name its present as 1 and 0 for rest. and red fields are the counters and flags should be the same. Hope it clarifies more about my problem. Main thing is sheet 2 and 3 should be integrated in such a way that every file has one row representation only. Dont bother how many fields. Regards

geniusgenie · ‎06-02-2017

Hi, I was wondering if someone can please help me? I have three datasets, all of them have one common column called file name. First dataset contains 2000 rows all unique rows (One row per filename). Second one has multiple rows per file name, some filenames have 7 rows and some more than 7. Similar for third dataset. I am looking to integrate second and third datasets in a way that both of these datasets have same one row per filename. Can anyone suggest how should I proceed for that? So, that I can merge both of these datasets into First dataset as features. Sample table Filename Category Type Col1 Col2 Col3 Col4 Co5 Flg Read Write Execute Align 1 1 LOAD 0x000000 00010000 00010000 1515c 1515c R E 1 0 1 0x10000 1 1 GNU_STACK 0x000000 00000000 00000000 00000 00000 RW 1 1 0 0x4 1 1 LOAD 0x01515c 0003515c 0003515c 00518 06874 RW 1 1 0 0x10000 2 1 LOAD 0x000000 08048000 08048000 0d0a0 0d0a0 R E 1 0 1 0x1000 2 1 GNU_STACK 0x000000 00000000 00000000 00000 00000 RW 1 1 0 0x4 2 1 LOAD 0x00d0a0 080560a0 080560a0 0031c 06800 RW 1 1 0 0x1000 3 2 LOAD 0x000000 00010000 00010000 1123c 1123c R E 1 0 1 0x10000 3 2 GNU_STACK 0x000000 00000000 00000000 00000 00000 RW 1 1 0 0x4 3 2 LOAD 0x01123c 0003123c 0003123c 004f4 06644 RW 1 1 0 0x10000 Regards

geniusgenie · ‎06-02-2017

Hi, I was wondering if someone can help me? I am using some classifiers and my dataset contains some variables with no correlation and some with high correlation. I am looking to know that how important classifiers like SVM, Neural Network, Decision Tree, Random Forest and Logistic Regression deal with highly correlated variables? Do they automatically reject them for classification or we have to manage ourself? If we have to manage ourself then how can I manage that ? In all of these classifiers how the selection and rejection of features takes place? Regards

geniusgenie · ‎06-01-2017

I will give it a go and lets hope it gives me something good. Thanks a lot

geniusgenie · ‎06-01-2017

Hi Reeza, thanks for your reply, How should I transform back? I am little confused on this. I was thinking to refer back to data dictionary which shows non standardized values of the columns and compare them with standardized ones but still its rough idea.. not sure about it.

geniusgenie · ‎06-01-2017

Hi, I was wondering if someone can clear my concepts about a question? I am using decision tree with standaridized data, in my previous that was whether I should used standardized data with decision tree or not, one of the members suggested that it wont hurt my model.... The problem I am facing is that few columns in both standardized or non standardized shape are harder to interpret. The difference standardization makes is to bring them in a range. Now atleast I know that my standardized data falls between 0 and 1. Only thing to worry is "How should I interpret the results in Tree algorithms" ?? Should I maintain data dictionary with both Unique standardized and Non Standardized values and compare them or some other recommended way? Regards

geniusgenie · ‎05-31-2017

Hi, Padraic, Thanks for your reply. I will follow the steps you have mentioned to see if these work for me. Regards

geniusgenie · ‎05-28-2017

Hi Foobarbaz, thanks for your reply, could you please tell me how can I run this code in EM? and do I need to attach my data partition or file import nodes with it??

geniusgenie · ‎05-27-2017

Hi, I wonder if anyone can help me? I am looking to apply following into my dataset , but cannot find many of these in the list of parameters. For example, I cannot find Hopfield Network, Ridge Regression etc. I am looking to integrate all of these, another example, I am able to find C4.5 in HP Forest node but not C5.0 I would appreciate, I can get help about these. Regards

geniusgenie · ‎05-26-2017

Hi Wendy, Thanks for sharing this wonderful article which really clarifies things but my question remains same if my dataset is already standardized would it be wrong to use random forest and decision tree or its ok. Plus, the dataset i have contains 5 colmuns which were difficult to be analysed without standardization for example all of them contain memory addresses like 0x00979876 etc. Only thing i did was to convert these 5 into decimal representation and standardized all of them. Other columns have normal categorical values. Hope this clarifies more. Regards

geniusgenie · ‎05-25-2017

Thanks for your reply Radhikha , but if we have already standardized it. Is it going to make any difference or going to produce wrong results?

Online Status	Offline
Date Last Visited	‎09-09-2017 10:31 AM

Re: How to check overfitting

Re: How to check overfitting

Re: How to check overfitting

How to check overfitting

Re: How to calculate f-score of classifiers

Re: Need an advice on Imbalance datasets

Re: Need an advice on Imbalance datasets

Re: Missing values in a column

feature selection methods in sas em

Missing values in a column

Re: How to do the feature extraction or selection in SAS Enterprise mi...

Re: How to calculate f-score of classifiers

Re: Correlated variables in classifiers

Re: Precision and recall scores in cutoff node

Re: Missing values in a column

Cross validation in machine learning algorithms

Re: dataset integration

Re: dataset integration

Re: dataset integration

Re: dataset integration

dataset integration

Correlated variables in classifiers

Re: How to interpret results when decision tree used with standardised...

Re: How to interpret results when decision tree used with standardised...

How to interpret results when decision tree used with standardised dat...

Re: decision stump and conditional decision trees

Re: How to perform n-gram analysis using Enterprise miner ?

How to implement Radial Basis Function Network

Re: Using standadized data for random forest and decision trees

Re: Using standadized data for random forest and decision trees