I have it on my personal computer and the folder was not backed up is there any chance I can recover it?    Due to the typical size and number of files in a SAS Enterprise Miner project, it is not practical to retain copies of those files in the software itself.   You will need to use appropriate backups for your type of operating system in order to protect against losing your work.      Hope this helps! Doug ... View more

Thanks Doug, do you also Standardize/Notmalize flag variables before using them in PROc factor?   There are scenarios when you might consider standardizing/normalizing variables and scenarios where you might not.   You just need to think about how the interpretations differ and decide which one makes more sense for your research question.   It is easy enough to run Factor Analysis both ways, one using the covariance matrix as the input and one using the correlation matrix as the input to get solutions using the raw data (covariance matrix) or standardized/normalized data (correlation matrix).   This same question about whether to standardize/normalize can be found when doing Principal Components Analysis (PCA) and predictive modeling.  In the end, methods which attempt to explain as much variability will be more influenced by variables with a larger amount of variability.  It doesn't inherently makes sense to weight a variable more heavily just because you altered the measurement units (e.g. from miles to inches), but neither does it makes sense to normalize variables which all have (theoretically) the same scale (e.g. survey questions which measure strength of response).        In the case of a survey where people are expressing their strength of agreement/satisfaction on some scale, it is natural that some questions will have greater variability than others and this could happen for a variety of reasons.      * some questions were poorly worded      * some areas were much more problematic than others      * the focus group itself has certain biases which might differ from the population   If my survey is trying to assess which factors are most important to the focus group, standardizing/normalizing the strength of agreement/satisfaction variables makes no sense because you are trying to identify what factors matter.   Normalizing in this situation effectively weighs every question equally regardless of how little variability the question represented.   On the other hand, if my goal is to create an overall metric that I plan to evaluate over time, failing to standardize makes the resulting scores less comparable since each question is potentially providing a different amount of influence on each solution.   In many cases, it might be that the survey instrument changes in response to reviewing results where certain questions had very little variability.   If my data is not naturally on the same scale (e.g. cost of a car in $, horsepower of a car, mileage of a car), then standardizing typically will make more sense but you also need to consider if your data represents the full range of values you want to consider or a narrow band of the population?     If I have certain variables which vary over a tiny portion of the population of interest while other variables span a large proportion of the population, standardizing effectively makes the variable which only varies over a tiny proportion of the range for the population of interest much more important giving it the same weight as variables which contain data much more representative of the whole population.     Neither approach is wrong, they just need to be interpreted differently, and considering which type of interpretation is of greater interest should help you decided whether or not to standardize and how to prepare your data as a result.   Hope this helps! Doug      ... View more

  So, if we look at my plot of CCC, that good performance will be, if there are more than 4 clusters ( more 2000 points)?     It is best to review the report since the CCC is just one way to evaluate a candidate number of clusters, and there are situation where the CCC might not be the best statstic to use.  The goal of clustering is typically to provide interpretable and/or usable results for your analysis needs.  Think of the CCC plot as recommending a range of cluster solutions that might be useful and you can then compare the competing solutions for which one best meets those needs.     When I see the CCC increasing slowly over the larger number of clusters, I would expect the additional splits to be pulling off clusters with small numbers of observations which can be useful if you are trying to isolate unusual potentially fraudulent cases but is not helpful if you are doing marketing where small clusters are not large enough to warrant special treatment.  Check multiple cluster solutions and choose what is best for your scenario.   Hope this helps! Doug ... View more

Please do help me with which option is recommended for me - Option-1,Option-2 or Option-3 in my case ? If Option-3, then is it recommended to use a oversampling rate of either 2% or 3% in order to increase the number of records to be modeled to something above 6487 ?   I don't know that one could say in most situations let alone a specific situation which approach will be most effective since this could differ depending on the data and the way in which the relationships among the variables might be changing over time.   The one that performs the best on the data you train on is less important than the one that works best in practice.  It is common to build multiple models and to pick which is the champion and which are the challengers based on their performance in practice.  I would typically say build multiple models and use your business judgement, needs for interpretation, statistical metrics of interest (each modeler might prefer different ones for different reasons), and go with it.  Over time, you should find yourself knowing which approach is best for your modeling situation and refitting a new model can occur more quickly.  Given the speed at which models can be built on modern software, you really shouldn't have to choose in advance.   Hope this helps! Doug ... View more

With a 5.77 oversampled rate, number of events = 374 and number of non-events=6113, a total of 6487 records. with a 70:30 split between TRAIN and VALIDATION , I can build my model on 4541 records and perform intime validation on 1946 records.   with a 70:30 split between TRAIN and VALIDATION , I can build my model on 42896 records and perform intime validation on 18383 records.   Some things that I would share based on my personal opinion and my limited knowledge of your modeling scenario:      *  I tend to think in terms of how many actual events I have rather than how many total records I have.   If I have 100 events and a 100 non-events, I have 200 observations.  If I have 100 events and 99,900 non-events, do I really have that much more information?  The signal in that case is so low (0.1%) that it would be difficult to have much confidence in any fitted model.      *  The total number of events (374) is so low that I would consider not even splitting the data in this situation.  Data Mining methods like partitioning assume that there are sufficient observations to represent the population in every partitioned data set.  Splitting 70/30 leaves barely over 100 events in validation.   Your data set might be better handle by classical statistical approaches given the limited data available.          *  I always fit a Tree to better understand structure in the data.  Trees can show you relationships in the data that you might not find from looking at a logistic regression model.   It can also show when you have specified a variable that you shouldn't have because it was actually an outcome variable (happens more often than you think).       If you need to build a model on the limited data you have, you should consider monitoring model performance.   The less signal the data has, the less models are truly viable.  Regardless of what model you fit, the confidence you can have in the usefulness of the model will be typically be greater when you have a larger number of target events, but that should not stop you from trying different methods and evaluating them with regards to your analysis objective.    It might also be that you have good and useful predictors and you don't actually need a lot more data.  The nice thing about modern software is that we can fit lots of different models and compare the results.  I don't ever start by choosing a single strategy.   Hope this helps! Cordially, Doug   ... View more

When doing a logistic regression, you can compute the Gains, Lift, %response, but also the ROC Curve and associated statistics of Gini and AUC. Do you have any advice on when to use each of these statistics to evaluate the quality of the model fit?   (Please note what follows is my personal opinion based on the modeling scenarios I have encountered, but opinions might vary greatly among modelers based on their own experience and objectives)   There are several things to consider when fitting a model, and how you want to use the model is critical to informing how you should select the model itself.      * Do you have specific objectives in how the model will be used?  For example, if my goal is to make a business decision on certain observations (e.g. choose a specific strategy for each observation/account), I might be more concerned about how effective my strategies are on select portions of the population than on how the model fits overall.  If I am dealing with a rare event, I am typically most interested in how the model performs on the (likely) small percent of the population on which I take action.  Lift/Gain are calculated at specific percentages of the population allowing me to evaluate strategy effectiveness at any given depth while Gini and AUC (area under the curve) assess overall model performance across the entire population. In some cases, however, interpretation is critical which begs another question.    * Do you need the model to be interpretable?   Should you have a need to interpret your fitted model, you will find yourself being forced to choose from a subset of modeling approaches in order to obtain this interpretation.  You should still consider more flexible modeling strategies, however, since the performance on these other models can give you an idea how much performance you are sacrificing for interpretability. Your choice in this case is more challenging since decision trees are often considered when interpretability is desired, but trees don't lend themselves to smoothly changing metrics.   Trees have a relatively small number of distinct predicted values from their terminal nodes, and every observation in the same node has the same predicted value.  At times, some have chosen to apply secondary models to try and better sort the observations within a node to overcome this.   If you have a Regression model, however, you can choose any distinct predicted value in the data as your cutoff value and there are typically not large blocks of observations with exactly the same score among the training observations.  Suppose the highest response rate occurred in terminal tree nodes with 12.3%, 6.4% and 4.8% of the data.   Lift charts are computed by evaluating bins (e.g. every 5% grouping) but this means the terminal nodes from the tree do not fit nicely into the bins.    * Do you have profit/cost information that should be considered?  You can specify decision weights for categorical target variable that allow you to evaluate the most profitable outcome.   In this case, many of the metrics you mentioned might be less critical than the projected revenue/profit from making certain decisions, even if those decisions are more likely to be false.  Suppose you had data with a 1% fraud rate.   A person 10 times as likely to be fraudulent still only has 10% chance of fraudulent even though there is a 90% chance of not being fraudulent. If the cost of fraud is in the tens of thousands and the cost of investigation is relatively small, you might consider investigating this person who is 90% likely to be non-fraudulent.   These types of considerations are not considered by Lift/Gain or Gini/AUC.   It is best to put think of how these different metrics can be balanced against the business objectives.  Are you more concerned about paying out huge fraud costs or are you more concerned about angering your non-fraudulent clientele and potentially losing business.  As you can see, choosing the best criterion can be difficult at times.   Hope this helps! Cordially, Doug  ... View more

Isn't the 200% the lift you have calculated and not the gain?   Lift can be computed by dividing the group response rate by the overall response rate.  In my previous example, if the group response rate is 12% and the the overall response rate is 4%,  Lift = 12% / 4% = 3 (no units on Lift) which is the same as 300%.   Had the group only had a 4% response rate, Lift = 4% / 4% = 1 (or 100%) but since the group does no better than the overall population, the Gain is 0 (essentially, Lift -1)   A lift of 4 would mean the group has a response rate which is 4 times the overall response rate but that is only 3 times better than the background rate so the gain would be 3 (or 300%).   Hope this helps! Cordially, Doug  ... View more

I  have built a model and also found the champion model. I  would like to test how model is performing. I have  four months data with the outcome. Now my question is how should I proceeded the validation of model by calculating  confusion matrix ( ppv, sensitivist etc )   Will you please walk me thru the process?   Validation of a model is typically done during the training process by using a holdout sample of data (called the "validation" data) to compare candidate models which were fit to the training data.  It is not uncommon to choose the candidate model which performed best on the validation data and then put that model into production for regular scoring purposes.     Unfortunately, models have a "shelf life" and tend to perform somewhat more poorly over time.  You can evaluate ongoing performance over time in a variety of ways but they all involve comparing recent model performance to how the model performed when it was first implemented to assess how that performance has changed over time.  As needed, you can build new "challenger" models to compete against previously implemented "champion" models to determine whether the "champion" model should be replaced by the "challenger" model for regular scoring purposes.    The approach to monitoring model performance depends on what tools you have available.  SAS offers products such as SAS Model Manager that assists with managing your models or you can do so using a more manual method but the basic outline is similar:   1 - Build the best possible model (the "champion" model) in light of your business objectives  2 - Implement the "champion" model into your production scoring. 3 - Monitor the performance of the "champion" model at regular intervals. 4 - Build new "challenger" models should the "champion" model performance become too poor and/or if the business objectives/climate change and/or if substantial time has passed  5 - Compare performance of the "challenger" and "champion" models to determine whether to retain or replace the previous champion model. 6 - Repeat steps 3-5 as needed   Hope this helps! Cordially, Doug   ... View more

My neural networks do not have AIC in their fit statistics (SAS EM 14.3, Interval target). What is a reason for that?   The AIC statistic is a "penalized" criterion which penalizes based on the number of parameters in a model.  The AIC is only computed for training data (not validate or test data) when the data can estimate the model parameters while retaining degrees of freedom for error.    Since Neural network models use the same predictors in multiple ways, they can potentially require more parameters than can be fit by the data leading to the node displaying negative degrees of freedom for error in the node results.  In these cases, no AIC will be reported.      Hope this helps! Cordially, Doug     ... View more