Hi, First of all, there is no over-sampling node in EM. I figure you meant Sample Node. The Sample Node has random, systematic, First, N, stratify... None of them allows you to change the ration between 1 and 0 on the target. The purpose of sampling is to take a subset, in one way or another, to represent the master source. The goal is to represent, not to alter. On the other hand, the matter of oversampling is to recompose a sample, therefore to alter, logically. Sampling Node often is used in situation like : The qualified model universe has 20 million observations. I need to take 5% sample to make it work in EM. In this sense, sampling really is not analytical/technical. But oversampling is every bit of analytics. In other words, the reason you run sampling should not overlap with that driving oversampling, although the act of oversampling per SE is sampling. Hope this helps? Thank you for using SAS. Best. Jason Xin ... View more

I agree your assessment about transmitting corporate data to SAS. It is doable. It just needs to go through some paperwork. Technology is ready. SAS TS has experience with it. But it depends on if your company grants it or not. Thanks. ... View more

It seems your max branch remains at 2. For the sake of testing, wonder if you can relax it to, say, 5, or 8. Also run StaExplore (if you have not) under Explore to profile the input variables. See if you have any significant /strong/dominating and highly categorical variables in the input set, like the most significant binary variable you mentioned. When I ran into situation like this, i often hold out (strong) categorical variables , and build a DT with continuous variables. Then throw back the hold out variables to check the impact on the model. ... View more

Hi, I have seen cases in the past where EM GBM performs in comparable speed with R integrated into the same flow, everything else roughly held equal. Yes, I have seen cases where GBM is slower than R. And vice Verse. So there is little general to infer or conclude. If I am to be very useful to you, in eventuality, I will have to see down in front of your data set and operations to help speed up, as I did several times in the past. Generally speaking, EM spends a lot of resource running the GUI operations, writing and rewriting code in the background, something that running R through the integration node does not entail. Often when one EM node runs this slow, it indicates the work space for the flow likely is running out of space. It is simply writing as it is swapping... This eventually is a SAS Management Console subject where one can try to relocate and optimize space management. If GUI operation does not appeal to you that much, you can try the underlying procedure TreeBOOST. If you go to Google.com, search for "Jason Xin, treeboost", you should quickly get to the full-fledged sample code I published years ago. Once you finish modeling using the procedure code, you can re-introduce the predicted value back to EM by using Model Import Node to align model comparison with other models you are building with EM GUI. Hope this helps. Thanks. Jason Xin ... View more

Hi, How many variables /observations are you trying with GBM node? Possible to run variable selection before GBM node? EM's HPFOREST node runs at least on multi-threading ability (SMP) of the CPUs; if you are configured to run on MPP (massively parallel processing) engaging, say, 32 or 48 computers, the speed and other performance are expected to be better than SMP. The GBM node is not supported to run on SMP or MPP. It runs on traditional single thread node; the node does not have a HP prefix in front of it (this is how you tell). The speed expectation, therefore, is not supposed to be in line with HPFOREST, besides inner algorithm difference. This is, on the other hand, in no way suggesting random forest runs faster than GBM, or vice verse. The latest SAS Viya sports a geninue in-memory GBM procedure and actions that scale on wide /table tables, a real big data. SAS is not expected to upgrade the existing EM GBM node to something like "HPGBM". Hope this helps? Thank you for using SAS. Best Regards Jason Xin ... View more

Hi, In EM, see attached picture. Once you load the data into EM, the YES group (in the picture) should be 1 in your case and NO group should be your 0 group. Count=999 should be your 518 and Count=967 should be your 252. To the right, in replace of 0.5081, enter 1. In replace of 0.4919, enter 2.055555556 (=518/252). In plain English, doing so you are telling EM to treat the 518 1 group as it is. And treat the 252 0 group as if there are 2.055555556*252~518. Logically. Hope this helps? Jason Xin ... View more

I forgot that attachment. Here it is. ... View more

Hi, lokendra_devang an_corecompete_ com,   If the client did the imputation before partition using EM Impute node, the imputation data steps should have been collected into the eventual scoreing piece using Score Node or Score Export Node. Automatically. I attach a PDF of 2 pages I took from EM user guide that shows which nodes generate steps that are automatically appended together by EM Score nodes automatically.    By design, nothing written under the EM SAS Code node is automatically picked into the final scoring equation by Score Node or Score Export Node.  Subtle is this: if and when set up properly (selecting the right tool type +options) the SAS Code Node runs as expected and produces as expected. This, however, does not give the license for the 'correct' stuff to be automatically incorporated into the final scoring equation. The reason behind this essentially is the price we pay for the great fliexibility affordedy the SAS Code Node. Almost anything and everything you have licensed, or anything you can stick into BASE Editor and make it run, can be inserted into the SAS Code node and run  successfully. If EM SAS Code node is designed to automatically write everything we put under Score Code Node, EM could very well end up like EG. And perform much worse than EG or any other SAS code writer.   In the past two years I have encountered two dozens SAS customers who have tried to insert thousands of data preparation codes into SAS Code Node and force EM to compile them into its final score code. The distinction is this: EM is a predictive modeling tool while one should use BASE, or Java, or Hive or C or whatever to prepare the input model universe as much as possible before inputing the data set into EM for modeling. The SAS Code Node is intended for 'mid-flow' or 'mid-stream' supplements that require facilities that are from beyond EM's built in scope. As a practical matter, it is much neater and much easier to inspect if you just lay bare thousands of your steps and procedure code in BASE or SAS Studio, instead of clicking through deep into the thick of EM Code.    For the purpose of completing your scoring piece, you don't need to copy over everything done related to validation, if any, in the Code node. Validation requires a present target variable which does not typically provide itself in your scoring process.    As for  your lop-sided segment% of 90%, I think you need to trace back down to make sure all the custom code the client analyst had injected into this EM process has been adequately recovered and brought back to your scoring job on hand, before you 'scream' again. The chance is: as you mentioned, the client analyst's pre-partition imputation logic is missing, then the 90% smells just like a category collapse due to the imputation piece being missing. I concur with you that if the client analyst did the imputation prior to the partitioning process in EM, the analyst may very well have done that on the data set before plugging the data set into EM to beging with.    Hope this help? Thank you for using SAS.    Best Regards Jason Xin ... View more