About zhongxiuliu

zhongxiuliu · ‎09-01-2021

I meant objective function and loss function are often used to describe the same thing 🙂 However, they are different from error function. Both function are error function (the error) + regularization (e.g., the squared or absolute value of weights; some people call it R1, R2; some people call it Lasso Ridge). The reason behind this is: if we just minimize the error, we can easily get a model with very big weights, which makes our activation function's slope really deep (a little change in x, causes big change in y); our model would overfit, unstable and sensitive to noise . Minimizing both error and the weights, makes our neural network less sensitive to noises in data, and more generalizable.

zhongxiuliu · ‎08-27-2021

1. epochs are group of iterations, representing when all the data has been used in updating weight. in stochastic gradient descent, each iteration we calculate derivatives and update weights, using only a small sample of all data. This way, when we have huge data, we can update quick. (For more information, google stochastic gradient descent.) If we have 500 data, each iteration we sample 50, then it take 10 iterations for all data to be used once. this 10 iterations, is an epoch. 2. validation score: the objective function, or loss. (error + regularization) 3. nthreads: is how many GPU devices you use for calculation, not related to the algorithm. SAS Viya use parallel computing, so its like how many computers you want to do the calculation. Larger number=> faster. But you can't have it larger than the available GPU your IT gives you. 4. stagnation: your understanding is correct. I suspect is keep going because of the objective function value is still going down, though the error stopped going down.

zhongxiuliu · ‎11-02-2020

sensitivity based pruning is another way to 'interpret' the model, similar to the concept of 'partial dependence plot'. By replacing all data's variable Xi's value to be a (or several) selected value(s), and applying score code to the new data, we see whether variable Xi is important enough for the target value prediction If there's redundant input, Xii, the score code would use Xii and make Xi looks important (overvalue). even though its a duplicated variable and should be filtered out in variable selection procedure. If there's interaction between Xi and Xii, Xi change Xii doesn't change kind of distort the original data's relationship, which would make Xi looks not that important (undervalue). I personally prefer use this method in the partial dependence plot department, for interpreting models instead of selecting inputs. Nowadays the data is often too complicated, having too much variables and not following specific distribution, to know interaction or redundant input ahead,

zhongxiuliu · ‎11-02-2020

initialization is how every neural network initialize. SAS will do it at the back end for you so you just need to understand the method, no need to implement it and it is not an optional method/approach to choose. preliminary training is setting in proc neural, to allow neural network initialize multiple times and select the best. Sometimes the randomness in initialization would make you trap in local minimum, and you want to give your model several times to restart to avoid this. Always try the preliminary training approach. Often your data is too complicated for you to know if you trapped in a local minimum.

zhongxiuliu · ‎11-02-2020

GANN is just like other neural network except that there's a direct connection between input and output (some people call it skipped layer neural network). So nominal and ordinal be encoded into numerical values and used just like in other neural networks. that would include binary inputs too

zhongxiuliu · ‎11-02-2020

that's a good approach to implement. For small neural network on data with well-defined features (often <=2 layers), I wouldn't go through the effort of sequential network construction. SAS Viya's PROC NNET auto-tuning is a better approach. With deep learning (deeplearn casl action set), drop-out is a also another better approach. You can look into both through online documentation.

zhongxiuliu · ‎11-02-2020

Both weight decay (regularization, adding weight to cost function to avoid overfitting) and early stopping (stop when performed the best at validation) are generally concept in machine learning models, not unique to neural networks. Many use both, and there's no clear preference which also works better than the other. Neural network is very easy to overfit the data, so i would recommend using both

zhongxiuliu · ‎11-02-2020

check online documentation in sas enterprise miner here https://support.sas.com/documentation/onlinedoc/miner/em151-143/emprcref.pdf mlogistic and softmax are the same function with different names.

Online Status	Offline
Date Last Visited	‎06-13-2023 01:04 PM

Re: LSTM with dltrain

Re: LSTM with dltrain

Re: Sensitivity Based Pruning

Re: Initialization Procedure vs Preliminary Training

Re: Use of nominal and ordinal inputs in GANN

Re: Use of Validation data with Sequential Network Construction

Re: Weight Decay vs Early Stopping

Re: Difference between MLogistic and SoftMax activation functions

2023 Customer Awards: Georgia Pacific - Curious Thinker

Re: LSTM with dltrain

Re: LSTM with dltrain

Re: LSTM with dltrain

Re: LSTM with dltrain

Re: Sensitivity Based Pruning

Re: Initialization Procedure vs Preliminary Training

Re: Use of nominal and ordinal inputs in GANN

Re: Use of Validation data with Sequential Network Construction

Re: Weight Decay vs Early Stopping

Re: Difference between MLogistic and SoftMax activation functions