About pvareschi

pvareschi · ‎11-04-2020

@PaigeMiller thank you for your explanations. First of all, my apologies, I misspelled my question...in the first paragraph I meant "what is the impact of obtaining parameter estimates with a high low precision". What I am struggling to get my head around is to link low precision to its effects on the estimates/predictions from a model, i.e. what does it mean in practice. For example, for simplicity, if we consider a simple linear regression model (i.e. 1 predictor), a low precision means that, although the coefficient estimate may not be biased (assuming regression model is appropriate), the coefficient estimate obtained from the sample used may be well off from the true population parameters (e.g. b-hat=5 while beta=2). In that scenario, it looks to me that predictions from that models would be inaccurate when applied to a new sample (assuming it was drawn/representative of the same underlying population). Is that correct?

pvareschi · ‎11-02-2020

Re: Predictive Modeling Using Logistic Regression When fitting a parametric model, such as Logistic Regression (or Linear Regression), what is the impact of obtaining parameter estimates with a high precision, in terms of developing a predictive model? (1) In other words, is it correct to say that if the precision is low (i.e. wide standard errors of coefficients) then the model is expected to perform poorly on new data because the fitted response surface may not be close enough (representative) of the true relationship in the underlying population? (2) When working with very large datasets, is it correct to say that precision becomes less of a concern because large samples always result in narrow standard errors? (3) Lastly, is multicollinearity a problem because of its adverse effect on the precision of the parameter estimates?

pvareschi · ‎10-04-2020

Re: Applied Analytics Using SAS Enterprise Miner -> Lesson 3: Introduction to Predictive Modeling Using SAS Enterprise Miner -> Model Complexity I am not sure I fully understand and appreciate the meaning and implications of the concepts of bias and variance as presented in the above lesson: 1. I understand that bias would occur with model underfitting, because, essentially, the model would not be flexible/complex enough to capture "the signal"; could bias occur with overfitting too? 2. What does "variance" refers to, when talking abot overfitted models? The lesson text reads: "[...] An overly complex model might be too flexible, which can lead to overfitting, that is, accommodating nuances of the random noise in the particular sample (high variance)" Does "high variance" refers to the fact that an overfitted model would produce highly variable/erratic predictions/results on a new data set (i.e. would not generalise well on new data)?

pvareschi · ‎08-05-2020

Someone from SAS may be able to provide a more accurate respose, but, as far as I know, the algorithm behind the Interactive Grouping Node uses a two-step approach for interval variables: 1. First, it "discretizes" the variables by creating groups, essentially transforming the variables from interval to nominal 2. Secondly, applies a Tree-based logic to find the optimal binning based on the groups from step (1) My understanding is the above approach is used only for computational efficiency reasons, because, in general, interval variables may have hundreds, if not, thousands of different values whihc would make it too computational intensive for a Tree algorithm to fully evaluate. Therefore, by carrying out a pre-binning step, you end up with far fewer categories which then can be optimised based on a Tree-like algorithm. Lastly, from my experience, unless you have a good reason for using "bucket", my advice is to always go for "quantile" (i.e. that should be the default approach unless, for some specific reason, you want to have groups defined by having the same width).

pvareschi · ‎06-17-2020

Of course, a high number of predictors would make things very difficult to manage. Just to give you an idea, I have worked on projects with between 500 and 1000 predictors! In those situations, I still find it useful to apply stratified sampling with only 3 maximum 5 key variables (i.e. inputs that are knows/expected to be very important, not only statistically, but from a business/problem context point of view). Something I forgot to mention in my first reply: before running statistical tests, I would recommend to compare the distribution visually, by using simple histrograms/bar charts...if the charts look similar then it might well be enough (i.e. you need to worry only when distributions are significantly different). Last thing: issues with sampling are most likely to affect predictors with skewed distributions, especially categorical ones. Therefore, before doing the sampling, make a note of categorical inputs with rare levels. Possible workarounds are: 1. Collapse levels before sampling 2. Increase the % sampled: instead of using a 50-50 split, you may want to take all 1% of events and 2% or 3% of non-events (i.e. 1:2 or 1:3 sampling ratio)

pvareschi · ‎06-17-2020

This is what I usually do when developing models within the financial sector (mainly scorecards to predict risk of default): 1. Use random sampling to select the 1% non-events; usually that guarantees you get a representative sample 2. As an alternative, you may want to stratify on important segmentation variables/factors. The choise of which variables to use depends on the context of your problem/analysis; for example: region, gender, age group, customer type... 3. Usually I do sampling with Proc SurveySelect 4. I usually check the 1% sample against the entire "non-events" population; one simple approach is to compare the distribution on key variables, based a similar rational as in step (2). Differences can be assessed either by using a Chi-square test or Information Value

pvareschi · ‎06-10-2020

Re: Advanced Predictive Modeling Using SAS 9.4 I would be grateful if someone could clarify the following points on the solution presented on the final Case Study for this module (references/times refers to the videos presenting the template solution): Q1. (Video "Advanced Model", time 9:00) Would it be possible to have a copy of the code used for the simulations to determine the optimal paramters for the Random Forest model? Q2. (Video "Advanced Model", time 10:42) Would it be possible to clarify what the Harold's approach is? The presented solution mentions that Harold's approach was described in the neural network course but I cannot find it in the course text for "Neural Network Modelling"

pvareschi · ‎06-09-2020

Topic 1 - How to handle missing values for predictive modelling: Overview of tools and techniques available Explanation of "when" and "why" a specific technique should be used (i.e. out of all the methods available, how do we decide which one to use in a specific situation and why?) Topic 2 - Practical examples of implementing/using Cross-Validation: When should cross-validation be used? Pros and cons Examples based on different modelling techniques (e.g. Regression vs Decision Trees) Enterprise Miner vs Enterprise Guide

pvareschi · ‎06-08-2020

Re: Applied Analytics Using SAS Enterprise Miner (this is a follow-up to a previous post) I would be grateful if someone could clarify the following points on the solution presented on the final Case Study for this module (references/times refers to the videos presenting the template solution): Q1. (Video "Exploring the Data", time 1:56) In the proposed solution, class variables with a mode percentage above 97% are excluded: how was 97% chosen? Is it based on a rule of thumb? Would it not make sense to take an even lower threshold, such as 95 or 90%, since highly skewed class predictors are likely to be related to very specific sub-populations therefore they may not help a lot in terms of creating a predictive model that generalises across a whole population (from experience, that is even more important and relevant when working with a binary target) Q2. (Video "Modifying the Data", time 0:24) The data is partitioned by taking a random sample stratified on all categorical variables: is that recommended and/or standard practice, even when there are many variables with potentially several levels? Would it not be enough to take a random sample with stratification based on the levels of the target variable only? Q3. (Video "Modifying the Data") Imputation of missing values: (a) (time 2:40) What is the purpose of defining the missing indicator variables as the number of variables with missing values? How are they used in the process/modelling? (b) (time 3:10) A threshold of "60% missing values" is used to decide whether an estimation or synthetic distribution approach should be used. In general, is it correct to say that an estimation approach is appropriate only when the percentage of missing values is not too large? (c) (time 3:25) What is the target variable used to assess the performance of the Decision Trees which are part of the data imputation process? Is it the original target variable? Q4. (Video "Building a Set of Predictive Models") (a) (time 0:39) Just to clarify, is the Variable Selection node set up to only affect interval variables (i.e. all categorical variables are left unchanged and set to Role=Input)? And those interval variables that are found to be non-significant, are set to rejected (thus not used by the following Regression nodes)? (b) (time 1:25) The two Decision Tree models are fitted with no input screening preceding them: in general, should we be worried about fitting Decision Tree models when there are too many inputs, or can they deal with any number of input variables? Said in other words, would a Decision Tree model benefit from applying an input screening before fitting the model?

pvareschi · ‎06-08-2020

Of course; would a standadisation of an interval target help improve prediction estimates, for instance, if the target had a highly skewed distribution?

pvareschi · ‎06-08-2020

Thank you for the link; however, that refers to the Enterprise Miner Reference Help (which I have already gone through!) which provides details on the nodes used in Enterprise Miner, but not the underlying procedures. The course text at page 4.4 seems to imply that a separate document is available with full details on those procedures: perhaps it is a "non official" documentation not available to the wider public?

pvareschi · ‎06-08-2020

Re: Predictive Modeling Using Logistic Regression (page 3.6-3.7 of course text) Q1. Given that Missing Indicator variables are created with the purpose of "capturing the relationship of missingness with the Target", in order to keep things manageable, would it not make sense to check immediately whether they are correlated with the target and if not, just drop them from the dataset, even before start applying inputs screening for redundancy and irrelevancy? Q2. How does the percentage of missingness in a variable impact the choice of what imputation method we should use? (i.e. synthetic vs distributional)? Q3. When would it be appropriate to use "random imputation" (i.e. based on the observed distribution of cases - this is value "Distribution" of Impute Node in Enterprise Miner)? Would it be when missing values do not have a relationship with the target? Q4. Should imputation of missing values for a categorical variable be done before or after collapsing the levels? Q5. How should ordinal variables be handled in terms of missing values? The point is that ordinal variables are between categorical and interval, therefore, replacing missing values with the mode may not always be appropriate; likewise, treating missing as a separate category would essentially make the variable a categorical one

pvareschi · ‎06-08-2020

Re: Predictive Modeling Using Logistic Regression Apologies if this may not be directly related to the topics covered in the course text (page 1.19). After splitting the data and identifying the best model based on the performance on the validation dataset, would it make sense to merge together the training and validation datasets and re-fit the chosen model on the full set of observations to obtain more accurate estimates of its parameters? Is this approach used in practice? If so, I can see how that would work for a regression or neural network model, however, what about decision trees? Even if the inputs to be used were constrained to those found by the initial fitting, the splitting points may actually change: would that be ok?

pvareschi · ‎06-08-2020

Re: Predictive Modeling Using Logistic Regression With regard to using Variable Clustering as a way of dealing with input redundancy (page 3.40 of course text): Q1. Does it make sense to include binary variables (including those from categorical variables) when running Variable Clustering: should I include missing indicators/? My concern is that those variables may negatively affect the way Proc Varclus defines the clusters, moreover, how should we interpret the resuls if, for instance, the dummy variables from a categorical input are spread across different clusters? I feel that categorical/binary variables, by their very nature, are better screened based on relevancy, using methods such as Chi-Square or Variable Importance from a Decision Tree. Q2. Is it not too restrictive to only select 1 variable from each cluster? In case, how would I select 2 variables from each cluster: would it make sense to pick those related to lowest and highest "1-R2” ratio?

pvareschi · ‎06-08-2020

Re: Predictive Modeling Using Logistic Regression Alongside Empirical Logit Plots (page 3.54 of course text), would it make sense to also use Partial Residual Plots as done for Neural Networks (see page 4-55 of Neural Network Modelling)? The point is that Empirical Logit Plots are based simply on the univariate relationship between an input and the target, without taking account of other inputs:Partial Residual Plots, on the other hand, should provide a clearer picture. Perhaps Empirical Logit Plots could be used as a first step to get an initial idea of what inputs may require a transformation and Partial Residual Plots should be used to refine the transformation as part of the final model fitting step.

Online Status	Offline
Date Last Visited	‎11-09-2020 06:42 AM

Re: Effect of precision of parameter estimates on predictions

Effect of precision of parameter estimates on predictions

Bias vs Variance

Re: Binning and Pre-Binning in Interactive Grouping

Re: Oversample: what about non-event sample?

Re: Oversample: what about non-event sample?

Questions on Case Study student attrition

Re: Ask the Expert Suggestion Box

Clarifications on solution to Case Study (transfer student project)

Re: Standardisation of target variable in HP Neural Node

Do Custom Formats Exist?

Re: Difference between Frequency and Weight variables (Visual Statisti...

Re: Binning and Pre-Binning in Interactive Grouping

Questions on Case Study student attrition

Re: Oversample: what about non-event sample?

Re: Oversample: what about non-event sample?

Re: Effect of precision of parameter estimates on predictions

Effect of precision of parameter estimates on predictions

Bias vs Variance

Re: Binning and Pre-Binning in Interactive Grouping

Re: Oversample: what about non-event sample?

Re: Oversample: what about non-event sample?

Questions on Case Study student attrition

Re: Ask the Expert Suggestion Box

Clarifications on solution to Case Study (transfer student project)

Re: Standardisation of target variable in HP Neural Node

Re: SAS Enterprise Miner User's Guide

Clarifications on handling missing values

Considerations on Data Splitting

Clarification on Variable Clustering

Empirical Logit vs Partial Residual Plots for Logistic Regression